{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Modelling Intrusion Detection: Analysis of a Feature Selection Mechanism\n",
    "\n",
    "## Method Description\n",
    "\n",
    "### Step 1: Data preprocessing:\n",
    "All features are made numerical using one-Hot-encoding. The features are scaled to avoid features with large values that may weigh too much in the results.\n",
    "\n",
    "### Step 2: Feature Selection:\n",
    "Eliminate redundant and irrelevant data by selecting a subset of relevant features that fully represents the given problem.\n",
    "Univariate feature selection with ANOVA F-test. This analyzes each feature individually to detemine the strength of the relationship between the feature and labels. Using SecondPercentile method (sklearn.feature_selection) to select features based on percentile of the highest scores. \n",
    "When this subset is found: Recursive Feature Elimination (RFE) is applied.\n",
    "\n",
    "### Step 4: Build the model:\n",
    "Decision tree model is built.\n",
    "\n",
    "### Step 5: Prediction & Evaluation (validation):\n",
    "Using the test data to make predictions of the model.\n",
    "Multiple scores are considered such as:accuracy score, recall, f-measure, confusion matrix.\n",
    "perform a 10-fold cross-validation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Version Check"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.19.2\n",
      "1.11.3\n",
      "3.6.0 |Anaconda 4.3.0 (64-bit)| (default, Dec 23 2016, 11:57:41) [MSC v.1900 64 bit (AMD64)]\n",
      "0.18.1\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import sys\n",
    "import sklearn\n",
    "print(pd.__version__)\n",
    "print(np.__version__)\n",
    "print(sys.version)\n",
    "print(sklearn.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load the Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dimensions of the Training set: (125973, 42)\n",
      "Dimensions of the Test set: (22544, 42)\n"
     ]
    }
   ],
   "source": [
    "# attach the column names to the dataset\n",
    "col_names = [\"duration\",\"protocol_type\",\"service\",\"flag\",\"src_bytes\",\n",
    "    \"dst_bytes\",\"land\",\"wrong_fragment\",\"urgent\",\"hot\",\"num_failed_logins\",\n",
    "    \"logged_in\",\"num_compromised\",\"root_shell\",\"su_attempted\",\"num_root\",\n",
    "    \"num_file_creations\",\"num_shells\",\"num_access_files\",\"num_outbound_cmds\",\n",
    "    \"is_host_login\",\"is_guest_login\",\"count\",\"srv_count\",\"serror_rate\",\n",
    "    \"srv_serror_rate\",\"rerror_rate\",\"srv_rerror_rate\",\"same_srv_rate\",\n",
    "    \"diff_srv_rate\",\"srv_diff_host_rate\",\"dst_host_count\",\"dst_host_srv_count\",\n",
    "    \"dst_host_same_srv_rate\",\"dst_host_diff_srv_rate\",\"dst_host_same_src_port_rate\",\n",
    "    \"dst_host_srv_diff_host_rate\",\"dst_host_serror_rate\",\"dst_host_srv_serror_rate\",\n",
    "    \"dst_host_rerror_rate\",\"dst_host_srv_rerror_rate\",\"label\"]\n",
    "\n",
    "# KDDTrain+_2.csv & KDDTest+_2.csv are the datafiles without the last column about the difficulty score\n",
    "# these have already been removed.\n",
    "df = pd.read_csv(\"KDDTrain+_2.csv\", header=None, names = col_names)\n",
    "df_test = pd.read_csv(\"KDDTest+_2.csv\", header=None, names = col_names)\n",
    "\n",
    "# shape, this gives the dimensions of the dataset\n",
    "print('Dimensions of the Training set:',df.shape)\n",
    "print('Dimensions of the Test set:',df_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sample view of the training dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>duration</th>\n",
       "      <th>protocol_type</th>\n",
       "      <th>service</th>\n",
       "      <th>flag</th>\n",
       "      <th>src_bytes</th>\n",
       "      <th>dst_bytes</th>\n",
       "      <th>land</th>\n",
       "      <th>wrong_fragment</th>\n",
       "      <th>urgent</th>\n",
       "      <th>hot</th>\n",
       "      <th>...</th>\n",
       "      <th>dst_host_srv_count</th>\n",
       "      <th>dst_host_same_srv_rate</th>\n",
       "      <th>dst_host_diff_srv_rate</th>\n",
       "      <th>dst_host_same_src_port_rate</th>\n",
       "      <th>dst_host_srv_diff_host_rate</th>\n",
       "      <th>dst_host_serror_rate</th>\n",
       "      <th>dst_host_srv_serror_rate</th>\n",
       "      <th>dst_host_rerror_rate</th>\n",
       "      <th>dst_host_srv_rerror_rate</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>tcp</td>\n",
       "      <td>ftp_data</td>\n",
       "      <td>SF</td>\n",
       "      <td>491</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>25</td>\n",
       "      <td>0.17</td>\n",
       "      <td>0.03</td>\n",
       "      <td>0.17</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.05</td>\n",
       "      <td>0.00</td>\n",
       "      <td>normal</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>udp</td>\n",
       "      <td>other</td>\n",
       "      <td>SF</td>\n",
       "      <td>146</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.60</td>\n",
       "      <td>0.88</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>normal</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>tcp</td>\n",
       "      <td>private</td>\n",
       "      <td>S0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>26</td>\n",
       "      <td>0.10</td>\n",
       "      <td>0.05</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>1.00</td>\n",
       "      <td>1.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>neptune</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>tcp</td>\n",
       "      <td>http</td>\n",
       "      <td>SF</td>\n",
       "      <td>232</td>\n",
       "      <td>8153</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>255</td>\n",
       "      <td>1.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.03</td>\n",
       "      <td>0.04</td>\n",
       "      <td>0.03</td>\n",
       "      <td>0.01</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.01</td>\n",
       "      <td>normal</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>tcp</td>\n",
       "      <td>http</td>\n",
       "      <td>SF</td>\n",
       "      <td>199</td>\n",
       "      <td>420</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>255</td>\n",
       "      <td>1.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>normal</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 42 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   duration protocol_type   service flag  src_bytes  dst_bytes  land  \\\n",
       "0         0           tcp  ftp_data   SF        491          0     0   \n",
       "1         0           udp     other   SF        146          0     0   \n",
       "2         0           tcp   private   S0          0          0     0   \n",
       "3         0           tcp      http   SF        232       8153     0   \n",
       "4         0           tcp      http   SF        199        420     0   \n",
       "\n",
       "   wrong_fragment  urgent  hot   ...     dst_host_srv_count  \\\n",
       "0               0       0    0   ...                     25   \n",
       "1               0       0    0   ...                      1   \n",
       "2               0       0    0   ...                     26   \n",
       "3               0       0    0   ...                    255   \n",
       "4               0       0    0   ...                    255   \n",
       "\n",
       "   dst_host_same_srv_rate  dst_host_diff_srv_rate  \\\n",
       "0                    0.17                    0.03   \n",
       "1                    0.00                    0.60   \n",
       "2                    0.10                    0.05   \n",
       "3                    1.00                    0.00   \n",
       "4                    1.00                    0.00   \n",
       "\n",
       "   dst_host_same_src_port_rate  dst_host_srv_diff_host_rate  \\\n",
       "0                         0.17                         0.00   \n",
       "1                         0.88                         0.00   \n",
       "2                         0.00                         0.00   \n",
       "3                         0.03                         0.04   \n",
       "4                         0.00                         0.00   \n",
       "\n",
       "   dst_host_serror_rate  dst_host_srv_serror_rate  dst_host_rerror_rate  \\\n",
       "0                  0.00                      0.00                  0.05   \n",
       "1                  0.00                      0.00                  0.00   \n",
       "2                  1.00                      1.00                  0.00   \n",
       "3                  0.03                      0.01                  0.00   \n",
       "4                  0.00                      0.00                  0.00   \n",
       "\n",
       "   dst_host_srv_rerror_rate    label  \n",
       "0                      0.00   normal  \n",
       "1                      0.00   normal  \n",
       "2                      0.00  neptune  \n",
       "3                      0.01   normal  \n",
       "4                      0.00   normal  \n",
       "\n",
       "[5 rows x 42 columns]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# first five rows\n",
    "df.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Statistical Summary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>duration</th>\n",
       "      <th>src_bytes</th>\n",
       "      <th>dst_bytes</th>\n",
       "      <th>land</th>\n",
       "      <th>wrong_fragment</th>\n",
       "      <th>urgent</th>\n",
       "      <th>hot</th>\n",
       "      <th>num_failed_logins</th>\n",
       "      <th>logged_in</th>\n",
       "      <th>num_compromised</th>\n",
       "      <th>...</th>\n",
       "      <th>dst_host_count</th>\n",
       "      <th>dst_host_srv_count</th>\n",
       "      <th>dst_host_same_srv_rate</th>\n",
       "      <th>dst_host_diff_srv_rate</th>\n",
       "      <th>dst_host_same_src_port_rate</th>\n",
       "      <th>dst_host_srv_diff_host_rate</th>\n",
       "      <th>dst_host_serror_rate</th>\n",
       "      <th>dst_host_srv_serror_rate</th>\n",
       "      <th>dst_host_rerror_rate</th>\n",
       "      <th>dst_host_srv_rerror_rate</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>125973.00000</td>\n",
       "      <td>1.259730e+05</td>\n",
       "      <td>1.259730e+05</td>\n",
       "      <td>125973.000000</td>\n",
       "      <td>125973.000000</td>\n",
       "      <td>125973.000000</td>\n",
       "      <td>125973.000000</td>\n",
       "      <td>125973.000000</td>\n",
       "      <td>125973.000000</td>\n",
       "      <td>125973.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>125973.000000</td>\n",
       "      <td>125973.000000</td>\n",
       "      <td>125973.000000</td>\n",
       "      <td>125973.000000</td>\n",
       "      <td>125973.000000</td>\n",
       "      <td>125973.000000</td>\n",
       "      <td>125973.000000</td>\n",
       "      <td>125973.000000</td>\n",
       "      <td>125973.000000</td>\n",
       "      <td>125973.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>287.14465</td>\n",
       "      <td>4.556674e+04</td>\n",
       "      <td>1.977911e+04</td>\n",
       "      <td>0.000198</td>\n",
       "      <td>0.022687</td>\n",
       "      <td>0.000111</td>\n",
       "      <td>0.204409</td>\n",
       "      <td>0.001222</td>\n",
       "      <td>0.395736</td>\n",
       "      <td>0.279250</td>\n",
       "      <td>...</td>\n",
       "      <td>182.148945</td>\n",
       "      <td>115.653005</td>\n",
       "      <td>0.521242</td>\n",
       "      <td>0.082951</td>\n",
       "      <td>0.148379</td>\n",
       "      <td>0.032542</td>\n",
       "      <td>0.284452</td>\n",
       "      <td>0.278485</td>\n",
       "      <td>0.118832</td>\n",
       "      <td>0.120240</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>2604.51531</td>\n",
       "      <td>5.870331e+06</td>\n",
       "      <td>4.021269e+06</td>\n",
       "      <td>0.014086</td>\n",
       "      <td>0.253530</td>\n",
       "      <td>0.014366</td>\n",
       "      <td>2.149968</td>\n",
       "      <td>0.045239</td>\n",
       "      <td>0.489010</td>\n",
       "      <td>23.942042</td>\n",
       "      <td>...</td>\n",
       "      <td>99.206213</td>\n",
       "      <td>110.702741</td>\n",
       "      <td>0.448949</td>\n",
       "      <td>0.188922</td>\n",
       "      <td>0.308997</td>\n",
       "      <td>0.112564</td>\n",
       "      <td>0.444784</td>\n",
       "      <td>0.445669</td>\n",
       "      <td>0.306557</td>\n",
       "      <td>0.319459</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>82.000000</td>\n",
       "      <td>10.000000</td>\n",
       "      <td>0.050000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>0.00000</td>\n",
       "      <td>4.400000e+01</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>255.000000</td>\n",
       "      <td>63.000000</td>\n",
       "      <td>0.510000</td>\n",
       "      <td>0.020000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>0.00000</td>\n",
       "      <td>2.760000e+02</td>\n",
       "      <td>5.160000e+02</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>255.000000</td>\n",
       "      <td>255.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.070000</td>\n",
       "      <td>0.060000</td>\n",
       "      <td>0.020000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>42908.00000</td>\n",
       "      <td>1.379964e+09</td>\n",
       "      <td>1.309937e+09</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>77.000000</td>\n",
       "      <td>5.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>7479.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>255.000000</td>\n",
       "      <td>255.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>8 rows × 38 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "           duration     src_bytes     dst_bytes           land  \\\n",
       "count  125973.00000  1.259730e+05  1.259730e+05  125973.000000   \n",
       "mean      287.14465  4.556674e+04  1.977911e+04       0.000198   \n",
       "std      2604.51531  5.870331e+06  4.021269e+06       0.014086   \n",
       "min         0.00000  0.000000e+00  0.000000e+00       0.000000   \n",
       "25%         0.00000  0.000000e+00  0.000000e+00       0.000000   \n",
       "50%         0.00000  4.400000e+01  0.000000e+00       0.000000   \n",
       "75%         0.00000  2.760000e+02  5.160000e+02       0.000000   \n",
       "max     42908.00000  1.379964e+09  1.309937e+09       1.000000   \n",
       "\n",
       "       wrong_fragment         urgent            hot  num_failed_logins  \\\n",
       "count   125973.000000  125973.000000  125973.000000      125973.000000   \n",
       "mean         0.022687       0.000111       0.204409           0.001222   \n",
       "std          0.253530       0.014366       2.149968           0.045239   \n",
       "min          0.000000       0.000000       0.000000           0.000000   \n",
       "25%          0.000000       0.000000       0.000000           0.000000   \n",
       "50%          0.000000       0.000000       0.000000           0.000000   \n",
       "75%          0.000000       0.000000       0.000000           0.000000   \n",
       "max          3.000000       3.000000      77.000000           5.000000   \n",
       "\n",
       "           logged_in  num_compromised            ...             \\\n",
       "count  125973.000000    125973.000000            ...              \n",
       "mean        0.395736         0.279250            ...              \n",
       "std         0.489010        23.942042            ...              \n",
       "min         0.000000         0.000000            ...              \n",
       "25%         0.000000         0.000000            ...              \n",
       "50%         0.000000         0.000000            ...              \n",
       "75%         1.000000         0.000000            ...              \n",
       "max         1.000000      7479.000000            ...              \n",
       "\n",
       "       dst_host_count  dst_host_srv_count  dst_host_same_srv_rate  \\\n",
       "count   125973.000000       125973.000000           125973.000000   \n",
       "mean       182.148945          115.653005                0.521242   \n",
       "std         99.206213          110.702741                0.448949   \n",
       "min          0.000000            0.000000                0.000000   \n",
       "25%         82.000000           10.000000                0.050000   \n",
       "50%        255.000000           63.000000                0.510000   \n",
       "75%        255.000000          255.000000                1.000000   \n",
       "max        255.000000          255.000000                1.000000   \n",
       "\n",
       "       dst_host_diff_srv_rate  dst_host_same_src_port_rate  \\\n",
       "count           125973.000000                125973.000000   \n",
       "mean                 0.082951                     0.148379   \n",
       "std                  0.188922                     0.308997   \n",
       "min                  0.000000                     0.000000   \n",
       "25%                  0.000000                     0.000000   \n",
       "50%                  0.020000                     0.000000   \n",
       "75%                  0.070000                     0.060000   \n",
       "max                  1.000000                     1.000000   \n",
       "\n",
       "       dst_host_srv_diff_host_rate  dst_host_serror_rate  \\\n",
       "count                125973.000000         125973.000000   \n",
       "mean                      0.032542              0.284452   \n",
       "std                       0.112564              0.444784   \n",
       "min                       0.000000              0.000000   \n",
       "25%                       0.000000              0.000000   \n",
       "50%                       0.000000              0.000000   \n",
       "75%                       0.020000              1.000000   \n",
       "max                       1.000000              1.000000   \n",
       "\n",
       "       dst_host_srv_serror_rate  dst_host_rerror_rate  \\\n",
       "count             125973.000000         125973.000000   \n",
       "mean                   0.278485              0.118832   \n",
       "std                    0.445669              0.306557   \n",
       "min                    0.000000              0.000000   \n",
       "25%                    0.000000              0.000000   \n",
       "50%                    0.000000              0.000000   \n",
       "75%                    1.000000              0.000000   \n",
       "max                    1.000000              1.000000   \n",
       "\n",
       "       dst_host_srv_rerror_rate  \n",
       "count             125973.000000  \n",
       "mean                   0.120240  \n",
       "std                    0.319459  \n",
       "min                    0.000000  \n",
       "25%                    0.000000  \n",
       "50%                    0.000000  \n",
       "75%                    0.000000  \n",
       "max                    1.000000  \n",
       "\n",
       "[8 rows x 38 columns]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Label Distribution of Training and Test set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Label distribution Training set:\n",
      "normal             67343\n",
      "neptune            41214\n",
      "satan               3633\n",
      "ipsweep             3599\n",
      "portsweep           2931\n",
      "smurf               2646\n",
      "nmap                1493\n",
      "back                 956\n",
      "teardrop             892\n",
      "warezclient          890\n",
      "pod                  201\n",
      "guess_passwd          53\n",
      "buffer_overflow       30\n",
      "warezmaster           20\n",
      "land                  18\n",
      "imap                  11\n",
      "rootkit               10\n",
      "loadmodule             9\n",
      "ftp_write              8\n",
      "multihop               7\n",
      "phf                    4\n",
      "perl                   3\n",
      "spy                    2\n",
      "Name: label, dtype: int64\n",
      "\n",
      "Label distribution Test set:\n",
      "normal             9711\n",
      "neptune            4657\n",
      "guess_passwd       1231\n",
      "mscan               996\n",
      "warezmaster         944\n",
      "apache2             737\n",
      "satan               735\n",
      "processtable        685\n",
      "smurf               665\n",
      "back                359\n",
      "snmpguess           331\n",
      "saint               319\n",
      "mailbomb            293\n",
      "snmpgetattack       178\n",
      "portsweep           157\n",
      "ipsweep             141\n",
      "httptunnel          133\n",
      "nmap                 73\n",
      "pod                  41\n",
      "buffer_overflow      20\n",
      "multihop             18\n",
      "named                17\n",
      "ps                   15\n",
      "sendmail             14\n",
      "rootkit              13\n",
      "xterm                13\n",
      "teardrop             12\n",
      "xlock                 9\n",
      "land                  7\n",
      "xsnoop                4\n",
      "ftp_write             3\n",
      "udpstorm              2\n",
      "perl                  2\n",
      "worm                  2\n",
      "loadmodule            2\n",
      "sqlattack             2\n",
      "phf                   2\n",
      "imap                  1\n",
      "Name: label, dtype: int64\n"
     ]
    }
   ],
   "source": [
    "print('Label distribution Training set:')\n",
    "print(df['label'].value_counts())\n",
    "print()\n",
    "print('Label distribution Test set:')\n",
    "print(df_test['label'].value_counts())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Step 1: Data preprocessing:\n",
    "One-Hot-Encoding (one-of-K) is used to to transform all categorical features into binary features. \n",
    "Requirement for One-Hot-encoding:\n",
    "\"The input to this transformer should be a matrix of integers, denoting the values taken on by categorical (discrete) features. The output will be a sparse matrix where each column corresponds to one possible value of one feature. It is assumed that input features take on values in the range [0, n_values).\"\n",
    "\n",
    "Therefore the features first need to be transformed with LabelEncoder, to transform every category to a number."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Identify categorical features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training set:\n",
      "Feature 'protocol_type' has 3 categories\n",
      "Feature 'service' has 70 categories\n",
      "Feature 'flag' has 11 categories\n",
      "Feature 'label' has 23 categories\n",
      "\n",
      "Distribution of categories in service:\n",
      "http        40338\n",
      "private     21853\n",
      "domain_u     9043\n",
      "smtp         7313\n",
      "ftp_data     6860\n",
      "Name: service, dtype: int64\n"
     ]
    }
   ],
   "source": [
    "# colums that are categorical and not binary yet: protocol_type (column 2), service (column 3), flag (column 4).\n",
    "# explore categorical features\n",
    "print('Training set:')\n",
    "for col_name in df.columns:\n",
    "    if df[col_name].dtypes == 'object' :\n",
    "        unique_cat = len(df[col_name].unique())\n",
    "        print(\"Feature '{col_name}' has {unique_cat} categories\".format(col_name=col_name, unique_cat=unique_cat))\n",
    "\n",
    "#see how distributed the feature service is, it is evenly distributed and therefore we need to make dummies for all.\n",
    "print()\n",
    "print('Distribution of categories in service:')\n",
    "print(df['service'].value_counts().sort_values(ascending=False).head())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test set:\n",
      "Feature 'protocol_type' has 3 categories\n",
      "Feature 'service' has 64 categories\n",
      "Feature 'flag' has 11 categories\n",
      "Feature 'label' has 38 categories\n"
     ]
    }
   ],
   "source": [
    "# Test set\n",
    "print('Test set:')\n",
    "for col_name in df_test.columns:\n",
    "    if df_test[col_name].dtypes == 'object' :\n",
    "        unique_cat = len(df_test[col_name].unique())\n",
    "        print(\"Feature '{col_name}' has {unique_cat} categories\".format(col_name=col_name, unique_cat=unique_cat))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Conclusion: Need to make dummies for all categories as the distribution is fairly even. In total: 3+70+11=84 dummies.\n",
    "### Comparing the results shows that the Test set has fewer categories (6), these need to be added as empty columns."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# LabelEncoder"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Insert categorical features into a 2D numpy array"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>protocol_type</th>\n",
       "      <th>service</th>\n",
       "      <th>flag</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>tcp</td>\n",
       "      <td>ftp_data</td>\n",
       "      <td>SF</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>udp</td>\n",
       "      <td>other</td>\n",
       "      <td>SF</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>tcp</td>\n",
       "      <td>private</td>\n",
       "      <td>S0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>tcp</td>\n",
       "      <td>http</td>\n",
       "      <td>SF</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>tcp</td>\n",
       "      <td>http</td>\n",
       "      <td>SF</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  protocol_type   service flag\n",
       "0           tcp  ftp_data   SF\n",
       "1           udp     other   SF\n",
       "2           tcp   private   S0\n",
       "3           tcp      http   SF\n",
       "4           tcp      http   SF"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.preprocessing import LabelEncoder,OneHotEncoder\n",
    "categorical_columns=['protocol_type', 'service', 'flag']\n",
    "# insert code to get a list of categorical columns into a variable, categorical_columns\n",
    "categorical_columns=['protocol_type', 'service', 'flag'] \n",
    " # Get the categorical values into a 2D numpy array\n",
    "df_categorical_values = df[categorical_columns]\n",
    "testdf_categorical_values = df_test[categorical_columns]\n",
    "df_categorical_values.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Make column names for dummies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Protocol_type_icmp', 'Protocol_type_tcp', 'Protocol_type_udp', 'service_IRC', 'service_X11', 'service_Z39_50', 'service_aol', 'service_auth', 'service_bgp', 'service_courier', 'service_csnet_ns', 'service_ctf', 'service_daytime', 'service_discard', 'service_domain', 'service_domain_u', 'service_echo', 'service_eco_i', 'service_ecr_i', 'service_efs', 'service_exec', 'service_finger', 'service_ftp', 'service_ftp_data', 'service_gopher', 'service_harvest', 'service_hostnames', 'service_http', 'service_http_2784', 'service_http_443', 'service_http_8001', 'service_imap4', 'service_iso_tsap', 'service_klogin', 'service_kshell', 'service_ldap', 'service_link', 'service_login', 'service_mtp', 'service_name', 'service_netbios_dgm', 'service_netbios_ns', 'service_netbios_ssn', 'service_netstat', 'service_nnsp', 'service_nntp', 'service_ntp_u', 'service_other', 'service_pm_dump', 'service_pop_2', 'service_pop_3', 'service_printer', 'service_private', 'service_red_i', 'service_remote_job', 'service_rje', 'service_shell', 'service_smtp', 'service_sql_net', 'service_ssh', 'service_sunrpc', 'service_supdup', 'service_systat', 'service_telnet', 'service_tftp_u', 'service_tim_i', 'service_time', 'service_urh_i', 'service_urp_i', 'service_uucp', 'service_uucp_path', 'service_vmnet', 'service_whois', 'flag_OTH', 'flag_REJ', 'flag_RSTO', 'flag_RSTOS0', 'flag_RSTR', 'flag_S0', 'flag_S1', 'flag_S2', 'flag_S3', 'flag_SF', 'flag_SH']\n"
     ]
    }
   ],
   "source": [
    "# protocol type\n",
    "unique_protocol=sorted(df.protocol_type.unique())\n",
    "string1 = 'Protocol_type_'\n",
    "unique_protocol2=[string1 + x for x in unique_protocol]\n",
    "# service\n",
    "unique_service=sorted(df.service.unique())\n",
    "string2 = 'service_'\n",
    "unique_service2=[string2 + x for x in unique_service]\n",
    "# flag\n",
    "unique_flag=sorted(df.flag.unique())\n",
    "string3 = 'flag_'\n",
    "unique_flag2=[string3 + x for x in unique_flag]\n",
    "# put together\n",
    "dumcols=unique_protocol2 + unique_service2 + unique_flag2\n",
    "print(dumcols)\n",
    "\n",
    "#do same for test set\n",
    "unique_service_test=sorted(df_test.service.unique())\n",
    "unique_service2_test=[string2 + x for x in unique_service_test]\n",
    "testdumcols=unique_protocol2 + unique_service2_test + unique_flag2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Transform categorical features into numbers using LabelEncoder()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "   protocol_type  service  flag\n",
      "0              1       20     9\n",
      "1              2       44     9\n",
      "2              1       49     5\n",
      "3              1       24     9\n",
      "4              1       24     9\n"
     ]
    }
   ],
   "source": [
    "df_categorical_values_enc=df_categorical_values.apply(LabelEncoder().fit_transform)\n",
    "print(df_categorical_values_enc.head())\n",
    "# test set\n",
    "testdf_categorical_values_enc=testdf_categorical_values.apply(LabelEncoder().fit_transform)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# One-Hot-Encoding"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Protocol_type_icmp</th>\n",
       "      <th>Protocol_type_tcp</th>\n",
       "      <th>Protocol_type_udp</th>\n",
       "      <th>service_IRC</th>\n",
       "      <th>service_X11</th>\n",
       "      <th>service_Z39_50</th>\n",
       "      <th>service_aol</th>\n",
       "      <th>service_auth</th>\n",
       "      <th>service_bgp</th>\n",
       "      <th>service_courier</th>\n",
       "      <th>...</th>\n",
       "      <th>flag_REJ</th>\n",
       "      <th>flag_RSTO</th>\n",
       "      <th>flag_RSTOS0</th>\n",
       "      <th>flag_RSTR</th>\n",
       "      <th>flag_S0</th>\n",
       "      <th>flag_S1</th>\n",
       "      <th>flag_S2</th>\n",
       "      <th>flag_S3</th>\n",
       "      <th>flag_SF</th>\n",
       "      <th>flag_SH</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 84 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   Protocol_type_icmp  Protocol_type_tcp  Protocol_type_udp  service_IRC  \\\n",
       "0                 0.0                1.0                0.0          0.0   \n",
       "1                 0.0                0.0                1.0          0.0   \n",
       "2                 0.0                1.0                0.0          0.0   \n",
       "3                 0.0                1.0                0.0          0.0   \n",
       "4                 0.0                1.0                0.0          0.0   \n",
       "\n",
       "   service_X11  service_Z39_50  service_aol  service_auth  service_bgp  \\\n",
       "0          0.0             0.0          0.0           0.0          0.0   \n",
       "1          0.0             0.0          0.0           0.0          0.0   \n",
       "2          0.0             0.0          0.0           0.0          0.0   \n",
       "3          0.0             0.0          0.0           0.0          0.0   \n",
       "4          0.0             0.0          0.0           0.0          0.0   \n",
       "\n",
       "   service_courier   ...     flag_REJ  flag_RSTO  flag_RSTOS0  flag_RSTR  \\\n",
       "0              0.0   ...          0.0        0.0          0.0        0.0   \n",
       "1              0.0   ...          0.0        0.0          0.0        0.0   \n",
       "2              0.0   ...          0.0        0.0          0.0        0.0   \n",
       "3              0.0   ...          0.0        0.0          0.0        0.0   \n",
       "4              0.0   ...          0.0        0.0          0.0        0.0   \n",
       "\n",
       "   flag_S0  flag_S1  flag_S2  flag_S3  flag_SF  flag_SH  \n",
       "0      0.0      0.0      0.0      0.0      1.0      0.0  \n",
       "1      0.0      0.0      0.0      0.0      1.0      0.0  \n",
       "2      1.0      0.0      0.0      0.0      0.0      0.0  \n",
       "3      0.0      0.0      0.0      0.0      1.0      0.0  \n",
       "4      0.0      0.0      0.0      0.0      1.0      0.0  \n",
       "\n",
       "[5 rows x 84 columns]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "enc = OneHotEncoder()\n",
    "df_categorical_values_encenc = enc.fit_transform(df_categorical_values_enc)\n",
    "df_cat_data = pd.DataFrame(df_categorical_values_encenc.toarray(),columns=dumcols)\n",
    "# test set\n",
    "testdf_categorical_values_encenc = enc.fit_transform(testdf_categorical_values_enc)\n",
    "testdf_cat_data = pd.DataFrame(testdf_categorical_values_encenc.toarray(),columns=testdumcols)\n",
    "\n",
    "df_cat_data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Add 6 missing categories from train set to test set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['service_red_i',\n",
       " 'service_harvest',\n",
       " 'service_http_2784',\n",
       " 'service_http_8001',\n",
       " 'service_urh_i',\n",
       " 'service_aol']"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trainservice=df['service'].tolist()\n",
    "testservice= df_test['service'].tolist()\n",
    "difference=list(set(trainservice) - set(testservice))\n",
    "string = 'service_'\n",
    "difference=[string + x for x in difference]\n",
    "difference"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(22544, 84)"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "for col in difference:\n",
    "    testdf_cat_data[col] = 0\n",
    "\n",
    "testdf_cat_data.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Join encoded categorical dataframe with the non-categorical dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(125973, 123)\n",
      "(22544, 123)\n"
     ]
    }
   ],
   "source": [
    "newdf=df.join(df_cat_data)\n",
    "newdf.drop('flag', axis=1, inplace=True)\n",
    "newdf.drop('protocol_type', axis=1, inplace=True)\n",
    "newdf.drop('service', axis=1, inplace=True)\n",
    "# test data\n",
    "newdf_test=df_test.join(testdf_cat_data)\n",
    "newdf_test.drop('flag', axis=1, inplace=True)\n",
    "newdf_test.drop('protocol_type', axis=1, inplace=True)\n",
    "newdf_test.drop('service', axis=1, inplace=True)\n",
    "print(newdf.shape)\n",
    "print(newdf_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Split Dataset into 4 datasets for every attack category\n",
    "## Rename every attack label: 0=normal, 1=DoS, 2=Probe, 3=R2L and 4=U2R.\n",
    "## Replace labels column with new labels column\n",
    "## Make new datasets\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0    0\n",
      "1    0\n",
      "2    1\n",
      "3    0\n",
      "4    0\n",
      "Name: label, dtype: int64\n"
     ]
    }
   ],
   "source": [
    "# take label column\n",
    "labeldf=newdf['label']\n",
    "labeldf_test=newdf_test['label']\n",
    "# change the label column\n",
    "newlabeldf=labeldf.replace({ 'normal' : 0, 'neptune' : 1 ,'back': 1, 'land': 1, 'pod': 1, 'smurf': 1, 'teardrop': 1,'mailbomb': 1, 'apache2': 1, 'processtable': 1, 'udpstorm': 1, 'worm': 1,\n",
    "                           'ipsweep' : 2,'nmap' : 2,'portsweep' : 2,'satan' : 2,'mscan' : 2,'saint' : 2\n",
    "                           ,'ftp_write': 3,'guess_passwd': 3,'imap': 3,'multihop': 3,'phf': 3,'spy': 3,'warezclient': 3,'warezmaster': 3,'sendmail': 3,'named': 3,'snmpgetattack': 3,'snmpguess': 3,'xlock': 3,'xsnoop': 3,'httptunnel': 3,\n",
    "                           'buffer_overflow': 4,'loadmodule': 4,'perl': 4,'rootkit': 4,'ps': 4,'sqlattack': 4,'xterm': 4})\n",
    "newlabeldf_test=labeldf_test.replace({ 'normal' : 0, 'neptune' : 1 ,'back': 1, 'land': 1, 'pod': 1, 'smurf': 1, 'teardrop': 1,'mailbomb': 1, 'apache2': 1, 'processtable': 1, 'udpstorm': 1, 'worm': 1,\n",
    "                           'ipsweep' : 2,'nmap' : 2,'portsweep' : 2,'satan' : 2,'mscan' : 2,'saint' : 2\n",
    "                           ,'ftp_write': 3,'guess_passwd': 3,'imap': 3,'multihop': 3,'phf': 3,'spy': 3,'warezclient': 3,'warezmaster': 3,'sendmail': 3,'named': 3,'snmpgetattack': 3,'snmpguess': 3,'xlock': 3,'xsnoop': 3,'httptunnel': 3,\n",
    "                           'buffer_overflow': 4,'loadmodule': 4,'perl': 4,'rootkit': 4,'ps': 4,'sqlattack': 4,'xterm': 4})\n",
    "# put the new label column back\n",
    "newdf['label'] = newlabeldf\n",
    "newdf_test['label'] = newlabeldf_test\n",
    "print(newdf['label'].head())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train:\n",
      "Dimensions of DoS: (113270, 123)\n",
      "Dimensions of Probe: (78999, 123)\n",
      "Dimensions of R2L: (68338, 123)\n",
      "Dimensions of U2R: (67395, 123)\n",
      "Test:\n",
      "Dimensions of DoS: (17171, 123)\n",
      "Dimensions of Probe: (12132, 123)\n",
      "Dimensions of R2L: (12596, 123)\n",
      "Dimensions of U2R: (9778, 123)\n"
     ]
    }
   ],
   "source": [
    "to_drop_DoS = [2,3,4]\n",
    "to_drop_Probe = [1,3,4]\n",
    "to_drop_R2L = [1,2,4]\n",
    "to_drop_U2R = [1,2,3]\n",
    "DoS_df=newdf[~newdf['label'].isin(to_drop_DoS)];\n",
    "Probe_df=newdf[~newdf['label'].isin(to_drop_Probe)];\n",
    "R2L_df=newdf[~newdf['label'].isin(to_drop_R2L)];\n",
    "U2R_df=newdf[~newdf['label'].isin(to_drop_U2R)];\n",
    "\n",
    "#test\n",
    "DoS_df_test=newdf_test[~newdf_test['label'].isin(to_drop_DoS)];\n",
    "Probe_df_test=newdf_test[~newdf_test['label'].isin(to_drop_Probe)];\n",
    "R2L_df_test=newdf_test[~newdf_test['label'].isin(to_drop_R2L)];\n",
    "U2R_df_test=newdf_test[~newdf_test['label'].isin(to_drop_U2R)];\n",
    "print('Train:')\n",
    "print('Dimensions of DoS:' ,DoS_df.shape)\n",
    "print('Dimensions of Probe:' ,Probe_df.shape)\n",
    "print('Dimensions of R2L:' ,R2L_df.shape)\n",
    "print('Dimensions of U2R:' ,U2R_df.shape)\n",
    "print('Test:')\n",
    "print('Dimensions of DoS:' ,DoS_df_test.shape)\n",
    "print('Dimensions of Probe:' ,Probe_df_test.shape)\n",
    "print('Dimensions of R2L:' ,R2L_df_test.shape)\n",
    "print('Dimensions of U2R:' ,U2R_df_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Step 2: Feature Scaling:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Split dataframes into X & Y\n",
    "# assign X as a dataframe of feautures and Y as a series of outcome variables\n",
    "X_DoS = DoS_df.drop('label',1)\n",
    "Y_DoS = DoS_df.label\n",
    "X_Probe = Probe_df.drop('label',1)\n",
    "Y_Probe = Probe_df.label\n",
    "X_R2L = R2L_df.drop('label',1)\n",
    "Y_R2L = R2L_df.label\n",
    "X_U2R = U2R_df.drop('label',1)\n",
    "Y_U2R = U2R_df.label\n",
    "# test set\n",
    "X_DoS_test = DoS_df_test.drop('label',1)\n",
    "Y_DoS_test = DoS_df_test.label\n",
    "X_Probe_test = Probe_df_test.drop('label',1)\n",
    "Y_Probe_test = Probe_df_test.label\n",
    "X_R2L_test = R2L_df_test.drop('label',1)\n",
    "Y_R2L_test = R2L_df_test.label\n",
    "X_U2R_test = U2R_df_test.drop('label',1)\n",
    "Y_U2R_test = U2R_df_test.label"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Save a list of feature names for later use (it is the same for every attack category). Column names are dropped at this stage."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "colNames=list(X_DoS)\n",
    "colNames_test=list(X_DoS_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Use StandardScaler() to scale the dataframes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn import preprocessing\n",
    "scaler1 = preprocessing.StandardScaler().fit(X_DoS)\n",
    "X_DoS=scaler1.transform(X_DoS) \n",
    "scaler2 = preprocessing.StandardScaler().fit(X_Probe)\n",
    "X_Probe=scaler2.transform(X_Probe) \n",
    "scaler3 = preprocessing.StandardScaler().fit(X_R2L)\n",
    "X_R2L=scaler3.transform(X_R2L) \n",
    "scaler4 = preprocessing.StandardScaler().fit(X_U2R)\n",
    "X_U2R=scaler4.transform(X_U2R) \n",
    "# test data\n",
    "scaler5 = preprocessing.StandardScaler().fit(X_DoS_test)\n",
    "X_DoS_test=scaler5.transform(X_DoS_test) \n",
    "scaler6 = preprocessing.StandardScaler().fit(X_Probe_test)\n",
    "X_Probe_test=scaler6.transform(X_Probe_test) \n",
    "scaler7 = preprocessing.StandardScaler().fit(X_R2L_test)\n",
    "X_R2L_test=scaler7.transform(X_R2L_test) \n",
    "scaler8 = preprocessing.StandardScaler().fit(X_U2R_test)\n",
    "X_U2R_test=scaler8.transform(X_U2R_test) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Check that the Standard Deviation is 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  0.  1.\n",
      "  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.\n",
      "  1.  1.  1.  1.  1.  1.  1.  1.  0.  1.  1.  1.  1.  1.  1.  1.  1.  1.\n",
      "  1.  1.  1.  1.  1.  1.  1.  1.  1.  0.  1.  1.  0.  1.  0.  1.  1.  1.\n",
      "  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  0.  1.  1.  1.\n",
      "  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.\n",
      "  1.  1.  1.  1.  1.  1.  0.  1.  1.  1.  1.  1.  1.  1.]\n"
     ]
    }
   ],
   "source": [
    "print(X_DoS.std(axis=0))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X_Probe.std(axis=0);\n",
    "X_R2L.std(axis=0);\n",
    "X_U2R.std(axis=0);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Step 3: Feature Selection:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1. Univariate Feature Selection using ANOVA F-test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\Cynthia\\Anaconda3\\lib\\site-packages\\sklearn\\feature_selection\\univariate_selection.py:113: UserWarning: Features [ 16  44  63  66  68  86 114] are constant.\n",
      "  UserWarning)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(113270, 13)"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#univariate feature selection with ANOVA F-test. using secondPercentile method, then RFE\n",
    "#Scikit-learn exposes feature selection routines as objects that implement the transform method\n",
    "#SelectPercentile: removes all but a user-specified highest scoring percentage of features\n",
    "#f_classif: ANOVA F-value between label/feature for classification tasks.\n",
    "from sklearn.feature_selection import SelectPercentile, f_classif\n",
    "np.seterr(divide='ignore', invalid='ignore');\n",
    "selector=SelectPercentile(f_classif, percentile=10)\n",
    "X_newDoS = selector.fit_transform(X_DoS,Y_DoS)\n",
    "X_newDoS.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get the features that were selected: DoS"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['logged_in',\n",
       " 'count',\n",
       " 'serror_rate',\n",
       " 'srv_serror_rate',\n",
       " 'same_srv_rate',\n",
       " 'dst_host_count',\n",
       " 'dst_host_srv_count',\n",
       " 'dst_host_same_srv_rate',\n",
       " 'dst_host_serror_rate',\n",
       " 'dst_host_srv_serror_rate',\n",
       " 'service_http',\n",
       " 'flag_S0',\n",
       " 'flag_SF']"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "true=selector.get_support()\n",
    "newcolindex_DoS=[i for i, x in enumerate(true) if x]\n",
    "newcolname_DoS=list( colNames[i] for i in newcolindex_DoS )\n",
    "newcolname_DoS"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\Cynthia\\Anaconda3\\lib\\site-packages\\sklearn\\feature_selection\\univariate_selection.py:113: UserWarning: Features [ 4 16] are constant.\n",
      "  UserWarning)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(78999, 13)"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_newProbe = selector.fit_transform(X_Probe,Y_Probe)\n",
    "X_newProbe.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get the features that were selected: Probe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['logged_in',\n",
       " 'rerror_rate',\n",
       " 'srv_rerror_rate',\n",
       " 'dst_host_srv_count',\n",
       " 'dst_host_diff_srv_rate',\n",
       " 'dst_host_same_src_port_rate',\n",
       " 'dst_host_srv_diff_host_rate',\n",
       " 'dst_host_rerror_rate',\n",
       " 'dst_host_srv_rerror_rate',\n",
       " 'Protocol_type_icmp',\n",
       " 'service_eco_i',\n",
       " 'service_private',\n",
       " 'flag_SF']"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "true=selector.get_support()\n",
    "newcolindex_Probe=[i for i, x in enumerate(true) if x]\n",
    "newcolname_Probe=list( colNames[i] for i in newcolindex_Probe )\n",
    "newcolname_Probe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\Cynthia\\Anaconda3\\lib\\site-packages\\sklearn\\feature_selection\\univariate_selection.py:113: UserWarning: Features [  4  16  43  44  46  47  48  49  50  51  54  57  58  62  63  64  66  67\n",
      "  68  70  71  72  73  74  76  77  78  79  80  81  82  83  86  87  89  92\n",
      "  93  96  98  99 100 107 108 109 110 114] are constant.\n",
      "  UserWarning)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(68338, 13)"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_newR2L = selector.fit_transform(X_R2L,Y_R2L)\n",
    "X_newR2L.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get the features that were selected: R2L"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['src_bytes',\n",
       " 'dst_bytes',\n",
       " 'hot',\n",
       " 'num_failed_logins',\n",
       " 'is_guest_login',\n",
       " 'dst_host_srv_count',\n",
       " 'dst_host_same_src_port_rate',\n",
       " 'dst_host_srv_diff_host_rate',\n",
       " 'service_ftp',\n",
       " 'service_ftp_data',\n",
       " 'service_http',\n",
       " 'service_imap4',\n",
       " 'flag_RSTO']"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "true=selector.get_support()\n",
    "newcolindex_R2L=[i for i, x in enumerate(true) if x]\n",
    "newcolname_R2L=list( colNames[i] for i in newcolindex_R2L)\n",
    "newcolname_R2L"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\Cynthia\\Anaconda3\\lib\\site-packages\\sklearn\\feature_selection\\univariate_selection.py:113: UserWarning: Features [  4  16  43  44  46  47  48  49  50  51  54  57  58  62  63  64  66  67\n",
      "  68  70  71  72  73  74  75  76  77  78  79  80  81  82  83  86  87  89\n",
      "  92  93  96  98  99 100 107 108 109 110 114] are constant.\n",
      "  UserWarning)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(67395, 13)"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_newU2R = selector.fit_transform(X_U2R,Y_U2R)\n",
    "X_newU2R.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get the features that were selected: U2R"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['urgent',\n",
       " 'hot',\n",
       " 'root_shell',\n",
       " 'num_file_creations',\n",
       " 'num_shells',\n",
       " 'srv_diff_host_rate',\n",
       " 'dst_host_count',\n",
       " 'dst_host_srv_count',\n",
       " 'dst_host_same_src_port_rate',\n",
       " 'dst_host_srv_diff_host_rate',\n",
       " 'service_ftp_data',\n",
       " 'service_http',\n",
       " 'service_telnet']"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "true=selector.get_support()\n",
    "newcolindex_U2R=[i for i, x in enumerate(true) if x]\n",
    "newcolname_U2R=list( colNames[i] for i in newcolindex_U2R)\n",
    "newcolname_U2R"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Summary of features selected by Univariate Feature Selection"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Features selected for DoS: ['logged_in', 'count', 'serror_rate', 'srv_serror_rate', 'same_srv_rate', 'dst_host_count', 'dst_host_srv_count', 'dst_host_same_srv_rate', 'dst_host_serror_rate', 'dst_host_srv_serror_rate', 'service_http', 'flag_S0', 'flag_SF']\n",
      "\n",
      "Features selected for Probe: ['logged_in', 'rerror_rate', 'srv_rerror_rate', 'dst_host_srv_count', 'dst_host_diff_srv_rate', 'dst_host_same_src_port_rate', 'dst_host_srv_diff_host_rate', 'dst_host_rerror_rate', 'dst_host_srv_rerror_rate', 'Protocol_type_icmp', 'service_eco_i', 'service_private', 'flag_SF']\n",
      "\n",
      "Features selected for R2L: ['src_bytes', 'dst_bytes', 'hot', 'num_failed_logins', 'is_guest_login', 'dst_host_srv_count', 'dst_host_same_src_port_rate', 'dst_host_srv_diff_host_rate', 'service_ftp', 'service_ftp_data', 'service_http', 'service_imap4', 'flag_RSTO']\n",
      "\n",
      "Features selected for U2R: ['urgent', 'hot', 'root_shell', 'num_file_creations', 'num_shells', 'srv_diff_host_rate', 'dst_host_count', 'dst_host_srv_count', 'dst_host_same_src_port_rate', 'dst_host_srv_diff_host_rate', 'service_ftp_data', 'service_http', 'service_telnet']\n"
     ]
    }
   ],
   "source": [
    "print('Features selected for DoS:',newcolname_DoS)\n",
    "print()\n",
    "print('Features selected for Probe:',newcolname_Probe)\n",
    "print()\n",
    "print('Features selected for R2L:',newcolname_R2L)\n",
    "print()\n",
    "print('Features selected for U2R:',newcolname_U2R)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The authors state that \"After obtaining the adequate number of features during the univariate selection process, a recursive feature elimination (RFE) was operated with the number of features passed as parameter to identify the features selected\". This either implies that RFE is only used for obtaining the features previously selected but also obtaining the rank. This use of RFE is however very redundant as the features selected can be obtained in another way (Done in this project). One can also not say that the features were selected by RFE, as it was not used for this. The quote could however also imply that only the number 13 from univariate feature selection was used. RFE is then used for feature selection trying to find the best 13 features. With this use of RFE one can actually say that it was used for feature selection. However the authors obtained different numbers of features for every attack category, 12 for DoS, 15 for Probe, 13 for R2L and 11 for U2R. This concludes that it is not clear what mechanism is used for feature selection. \n",
    "\n",
    "## To procede with the data mining, the second option is considered as this uses RFE. From now on the number of features for every attack category is 13."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2. Recursive Feature Elimination for feature ranking (Option 1: get importance from previous selected)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "DoS Features sorted by their rank:\n",
      "[(1, 'same_srv_rate'), (2, 'count'), (3, 'flag_SF'), (4, 'dst_host_serror_rate'), (5, 'dst_host_same_srv_rate'), (6, 'dst_host_srv_count'), (7, 'dst_host_count'), (8, 'logged_in'), (9, 'serror_rate'), (10, 'dst_host_srv_serror_rate'), (11, 'srv_serror_rate'), (12, 'service_http'), (13, 'flag_S0')]\n"
     ]
    }
   ],
   "source": [
    "from sklearn.feature_selection import RFE\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "# Create a decision tree classifier. By convention, clf means 'classifier'\n",
    "clf = DecisionTreeClassifier(random_state=0)\n",
    "\n",
    "#rank all features, i.e continue the elimination until the last one\n",
    "rfe = RFE(clf, n_features_to_select=1)\n",
    "rfe.fit(X_newDoS, Y_DoS)\n",
    "print (\"DoS Features sorted by their rank:\")\n",
    "print (sorted(zip(map(lambda x: round(x, 4), rfe.ranking_), newcolname_DoS)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Probe Features sorted by their rank:\n",
      "[(1, 'dst_host_same_src_port_rate'), (2, 'dst_host_srv_count'), (3, 'dst_host_rerror_rate'), (4, 'service_private'), (5, 'logged_in'), (6, 'dst_host_diff_srv_rate'), (7, 'dst_host_srv_diff_host_rate'), (8, 'flag_SF'), (9, 'service_eco_i'), (10, 'rerror_rate'), (11, 'Protocol_type_icmp'), (12, 'dst_host_srv_rerror_rate'), (13, 'srv_rerror_rate')]\n"
     ]
    }
   ],
   "source": [
    "rfe.fit(X_newProbe, Y_Probe)\n",
    "print (\"Probe Features sorted by their rank:\")\n",
    "print (sorted(zip(map(lambda x: round(x, 4), rfe.ranking_), newcolname_Probe)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "R2L Features sorted by their rank:\n",
      "[(1, 'src_bytes'), (2, 'dst_bytes'), (3, 'hot'), (4, 'dst_host_srv_diff_host_rate'), (5, 'service_ftp_data'), (6, 'dst_host_same_src_port_rate'), (7, 'dst_host_srv_count'), (8, 'num_failed_logins'), (9, 'service_imap4'), (10, 'is_guest_login'), (11, 'service_ftp'), (12, 'flag_RSTO'), (13, 'service_http')]\n"
     ]
    }
   ],
   "source": [
    "rfe.fit(X_newR2L, Y_R2L)\n",
    " \n",
    "print (\"R2L Features sorted by their rank:\")\n",
    "print (sorted(zip(map(lambda x: round(x, 4), rfe.ranking_), newcolname_R2L)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "U2R Features sorted by their rank:\n",
      "[(1, 'hot'), (2, 'dst_host_srv_count'), (3, 'dst_host_count'), (4, 'root_shell'), (5, 'num_shells'), (6, 'service_ftp_data'), (7, 'dst_host_srv_diff_host_rate'), (8, 'num_file_creations'), (9, 'dst_host_same_src_port_rate'), (10, 'service_telnet'), (11, 'srv_diff_host_rate'), (12, 'service_http'), (13, 'urgent')]\n"
     ]
    }
   ],
   "source": [
    "rfe.fit(X_newU2R, Y_U2R)\n",
    " \n",
    "print (\"U2R Features sorted by their rank:\")\n",
    "print (sorted(zip(map(lambda x: round(x, 4), rfe.ranking_), newcolname_U2R)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2. Recursive Feature Elimination, select 13 features each of 122 (Option 2: get 13 best features from 122 from RFE)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from sklearn.feature_selection import RFE\n",
    "clf = DecisionTreeClassifier(random_state=0)\n",
    "rfe = RFE(estimator=clf, n_features_to_select=13, step=1)\n",
    "rfe.fit(X_DoS, Y_DoS)\n",
    "X_rfeDoS=rfe.transform(X_DoS)\n",
    "true=rfe.support_\n",
    "rfecolindex_DoS=[i for i, x in enumerate(true) if x]\n",
    "rfecolname_DoS=list(colNames[i] for i in rfecolindex_DoS)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "rfe.fit(X_Probe, Y_Probe)\n",
    "X_rfeProbe=rfe.transform(X_Probe)\n",
    "true=rfe.support_\n",
    "rfecolindex_Probe=[i for i, x in enumerate(true) if x]\n",
    "rfecolname_Probe=list(colNames[i] for i in rfecolindex_Probe)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "rfe.fit(X_R2L, Y_R2L)\n",
    "X_rfeR2L=rfe.transform(X_R2L)\n",
    "true=rfe.support_\n",
    "rfecolindex_R2L=[i for i, x in enumerate(true) if x]\n",
    "rfecolname_R2L=list(colNames[i] for i in rfecolindex_R2L)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "rfe.fit(X_U2R, Y_U2R)\n",
    "X_rfeU2R=rfe.transform(X_U2R)\n",
    "true=rfe.support_\n",
    "rfecolindex_U2R=[i for i, x in enumerate(true) if x]\n",
    "rfecolname_U2R=list(colNames[i] for i in rfecolindex_U2R)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Summary of features selected by RFE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Features selected for DoS: ['src_bytes', 'dst_bytes', 'wrong_fragment', 'num_compromised', 'same_srv_rate', 'diff_srv_rate', 'dst_host_count', 'dst_host_same_srv_rate', 'dst_host_serror_rate', 'dst_host_srv_serror_rate', 'service_ecr_i', 'flag_RSTR', 'flag_S0']\n",
      "\n",
      "Features selected for Probe: ['src_bytes', 'dst_bytes', 'rerror_rate', 'dst_host_same_srv_rate', 'dst_host_diff_srv_rate', 'dst_host_same_src_port_rate', 'dst_host_rerror_rate', 'service_finger', 'service_ftp_data', 'service_http', 'service_private', 'service_smtp', 'service_telnet']\n",
      "\n",
      "Features selected for R2L: ['duration', 'src_bytes', 'dst_bytes', 'hot', 'num_failed_logins', 'num_access_files', 'dst_host_count', 'dst_host_srv_count', 'dst_host_same_srv_rate', 'dst_host_same_src_port_rate', 'dst_host_srv_diff_host_rate', 'service_ftp_data', 'service_imap4']\n",
      "\n",
      "Features selected for U2R: ['duration', 'src_bytes', 'dst_bytes', 'hot', 'root_shell', 'num_file_creations', 'num_shells', 'srv_count', 'dst_host_count', 'dst_host_same_srv_rate', 'dst_host_srv_diff_host_rate', 'service_ftp_data', 'service_other']\n"
     ]
    }
   ],
   "source": [
    "print('Features selected for DoS:',rfecolname_DoS)\n",
    "print()\n",
    "print('Features selected for Probe:',rfecolname_Probe)\n",
    "print()\n",
    "print('Features selected for R2L:',rfecolname_R2L)\n",
    "print()\n",
    "print('Features selected for U2R:',rfecolname_U2R)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(113270, 13)\n",
      "(78999, 13)\n",
      "(68338, 13)\n",
      "(67395, 13)\n"
     ]
    }
   ],
   "source": [
    "print(X_rfeDoS.shape)\n",
    "print(X_rfeProbe.shape)\n",
    "print(X_rfeR2L.shape)\n",
    "print(X_rfeU2R.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Step 4: Build the model:\n",
    "### Classifier is trained for all features and for reduced features, for later comparison.\n",
    "#### The classifier model itself is stored in the clf variable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DecisionTreeClassifier(class_weight=None, criterion='gini', max_depth=None,\n",
       "            max_features=None, max_leaf_nodes=None,\n",
       "            min_impurity_split=1e-07, min_samples_leaf=1,\n",
       "            min_samples_split=2, min_weight_fraction_leaf=0.0,\n",
       "            presort=False, random_state=0, splitter='best')"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# all features\n",
    "clf_DoS=DecisionTreeClassifier(random_state=0)\n",
    "clf_Probe=DecisionTreeClassifier(random_state=0)\n",
    "clf_R2L=DecisionTreeClassifier(random_state=0)\n",
    "clf_U2R=DecisionTreeClassifier(random_state=0)\n",
    "clf_DoS.fit(X_DoS, Y_DoS)\n",
    "clf_Probe.fit(X_Probe, Y_Probe)\n",
    "clf_R2L.fit(X_R2L, Y_R2L)\n",
    "clf_U2R.fit(X_U2R, Y_U2R)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DecisionTreeClassifier(class_weight=None, criterion='gini', max_depth=None,\n",
       "            max_features=None, max_leaf_nodes=None,\n",
       "            min_impurity_split=1e-07, min_samples_leaf=1,\n",
       "            min_samples_split=2, min_weight_fraction_leaf=0.0,\n",
       "            presort=False, random_state=0, splitter='best')"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# selected features\n",
    "clf_rfeDoS=DecisionTreeClassifier(random_state=0)\n",
    "clf_rfeProbe=DecisionTreeClassifier(random_state=0)\n",
    "clf_rfeR2L=DecisionTreeClassifier(random_state=0)\n",
    "clf_rfeU2R=DecisionTreeClassifier(random_state=0)\n",
    "clf_rfeDoS.fit(X_rfeDoS, Y_DoS)\n",
    "clf_rfeProbe.fit(X_rfeProbe, Y_Probe)\n",
    "clf_rfeR2L.fit(X_rfeR2L, Y_R2L)\n",
    "clf_rfeU2R.fit(X_rfeU2R, Y_U2R)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Step 5: Prediction & Evaluation (validation):"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Using all Features for each category"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Confusion Matrices\n",
    "## DoS"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1, 1, 0, ..., 0, 0, 0], dtype=int64)"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Apply the classifier we trained to the test data (which it has never seen before)\n",
    "clf_DoS.predict(X_DoS_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 0.,  1.],\n",
       "       [ 0.,  1.],\n",
       "       [ 1.,  0.],\n",
       "       [ 1.,  0.],\n",
       "       [ 1.,  0.],\n",
       "       [ 1.,  0.],\n",
       "       [ 1.,  0.],\n",
       "       [ 0.,  1.],\n",
       "       [ 1.,  0.],\n",
       "       [ 1.,  0.]])"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# View the predicted probabilities of the first 10 observations\n",
    "clf_DoS.predict_proba(X_DoS_test)[0:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>Predicted attacks</th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Actual attacks</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>9499</td>\n",
       "      <td>212</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2830</td>\n",
       "      <td>4630</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Predicted attacks     0     1\n",
       "Actual attacks               \n",
       "0                  9499   212\n",
       "1                  2830  4630"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Y_DoS_pred=clf_DoS.predict(X_DoS_test)\n",
    "# Create confusion matrix\n",
    "pd.crosstab(Y_DoS_test, Y_DoS_pred, rownames=['Actual attacks'], colnames=['Predicted attacks'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Probe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>Predicted attacks</th>\n",
       "      <th>0</th>\n",
       "      <th>2</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Actual attacks</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2337</td>\n",
       "      <td>7374</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>212</td>\n",
       "      <td>2209</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Predicted attacks     0     2\n",
       "Actual attacks               \n",
       "0                  2337  7374\n",
       "2                   212  2209"
      ]
     },
     "execution_count": 88,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Y_Probe_pred=clf_Probe.predict(X_Probe_test)\n",
    "# Create confusion matrix\n",
    "pd.crosstab(Y_Probe_test, Y_Probe_pred, rownames=['Actual attacks'], colnames=['Predicted attacks'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## R2L"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>Predicted attacks</th>\n",
       "      <th>0</th>\n",
       "      <th>3</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Actual attacks</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>9707</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2573</td>\n",
       "      <td>312</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Predicted attacks     0    3\n",
       "Actual attacks              \n",
       "0                  9707    4\n",
       "3                  2573  312"
      ]
     },
     "execution_count": 89,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Y_R2L_pred=clf_R2L.predict(X_R2L_test)\n",
    "# Create confusion matrix\n",
    "pd.crosstab(Y_R2L_test, Y_R2L_pred, rownames=['Actual attacks'], colnames=['Predicted attacks'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## U2R"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>Predicted attacks</th>\n",
       "      <th>0</th>\n",
       "      <th>4</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Actual attacks</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>9703</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>60</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Predicted attacks     0  4\n",
       "Actual attacks            \n",
       "0                  9703  8\n",
       "4                    60  7"
      ]
     },
     "execution_count": 91,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Y_U2R_pred=clf_U2R.predict(X_U2R_test)\n",
    "# Create confusion matrix\n",
    "pd.crosstab(Y_U2R_test, Y_U2R_pred, rownames=['Actual attacks'], colnames=['Predicted attacks'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Cross Validation: Accuracy, Precision, Recall, F-measure"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## DoS"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.99639 (+/- 0.00341)\n",
      "Precision: 0.99505 (+/- 0.00477)\n",
      "Recall: 0.99665 (+/- 0.00483)\n",
      "F-measure: 0.99585 (+/- 0.00392)\n"
     ]
    }
   ],
   "source": [
    "from sklearn.model_selection import cross_val_score\n",
    "from sklearn import metrics\n",
    "accuracy = cross_val_score(clf_DoS, X_DoS_test, Y_DoS_test, cv=10, scoring='accuracy')\n",
    "print(\"Accuracy: %0.5f (+/- %0.5f)\" % (accuracy.mean(), accuracy.std() * 2))\n",
    "precision = cross_val_score(clf_DoS, X_DoS_test, Y_DoS_test, cv=10, scoring='precision')\n",
    "print(\"Precision: %0.5f (+/- %0.5f)\" % (precision.mean(), precision.std() * 2))\n",
    "recall = cross_val_score(clf_DoS, X_DoS_test, Y_DoS_test, cv=10, scoring='recall')\n",
    "print(\"Recall: %0.5f (+/- %0.5f)\" % (recall.mean(), recall.std() * 2))\n",
    "f = cross_val_score(clf_DoS, X_DoS_test, Y_DoS_test, cv=10, scoring='f1')\n",
    "print(\"F-measure: %0.5f (+/- %0.5f)\" % (f.mean(), f.std() * 2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Probe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.99571 (+/- 0.00328)\n",
      "Precision: 0.99391 (+/- 0.00684)\n",
      "Recall: 0.99267 (+/- 0.00405)\n",
      "F-measure: 0.99329 (+/- 0.00512)\n"
     ]
    }
   ],
   "source": [
    "accuracy = cross_val_score(clf_Probe, X_Probe_test, Y_Probe_test, cv=10, scoring='accuracy')\n",
    "print(\"Accuracy: %0.5f (+/- %0.5f)\" % (accuracy.mean(), accuracy.std() * 2))\n",
    "precision = cross_val_score(clf_Probe, X_Probe_test, Y_Probe_test, cv=10, scoring='precision_macro')\n",
    "print(\"Precision: %0.5f (+/- %0.5f)\" % (precision.mean(), precision.std() * 2))\n",
    "recall = cross_val_score(clf_Probe, X_Probe_test, Y_Probe_test, cv=10, scoring='recall_macro')\n",
    "print(\"Recall: %0.5f (+/- %0.5f)\" % (recall.mean(), recall.std() * 2))\n",
    "f = cross_val_score(clf_Probe, X_Probe_test, Y_Probe_test, cv=10, scoring='f1_macro')\n",
    "print(\"F-measure: %0.5f (+/- %0.5f)\" % (f.mean(), f.std() * 2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## R2L"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.97952 (+/- 0.00984)\n",
      "Precision: 0.97216 (+/- 0.01610)\n",
      "Recall: 0.96978 (+/- 0.01337)\n",
      "F-measure: 0.97093 (+/- 0.01388)\n"
     ]
    }
   ],
   "source": [
    "accuracy = cross_val_score(clf_R2L, X_R2L_test, Y_R2L_test, cv=10, scoring='accuracy')\n",
    "print(\"Accuracy: %0.5f (+/- %0.5f)\" % (accuracy.mean(), accuracy.std() * 2))\n",
    "precision = cross_val_score(clf_R2L, X_R2L_test, Y_R2L_test, cv=10, scoring='precision_macro')\n",
    "print(\"Precision: %0.5f (+/- %0.5f)\" % (precision.mean(), precision.std() * 2))\n",
    "recall = cross_val_score(clf_R2L, X_R2L_test, Y_R2L_test, cv=10, scoring='recall_macro')\n",
    "print(\"Recall: %0.5f (+/- %0.5f)\" % (recall.mean(), recall.std() * 2))\n",
    "f = cross_val_score(clf_R2L, X_R2L_test, Y_R2L_test, cv=10, scoring='f1_macro')\n",
    "print(\"F-measure: %0.5f (+/- %0.5f)\" % (f.mean(), f.std() * 2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## U2R"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.99663 (+/- 0.00259)\n",
      "Precision: 0.86481 (+/- 0.08952)\n",
      "Recall: 0.91672 (+/- 0.10661)\n",
      "F-measure: 0.88628 (+/- 0.07462)\n"
     ]
    }
   ],
   "source": [
    "accuracy = cross_val_score(clf_U2R, X_U2R_test, Y_U2R_test, cv=10, scoring='accuracy')\n",
    "print(\"Accuracy: %0.5f (+/- %0.5f)\" % (accuracy.mean(), accuracy.std() * 2))\n",
    "precision = cross_val_score(clf_U2R, X_U2R_test, Y_U2R_test, cv=10, scoring='precision_macro')\n",
    "print(\"Precision: %0.5f (+/- %0.5f)\" % (precision.mean(), precision.std() * 2))\n",
    "recall = cross_val_score(clf_U2R, X_U2R_test, Y_U2R_test, cv=10, scoring='recall_macro')\n",
    "print(\"Recall: %0.5f (+/- %0.5f)\" % (recall.mean(), recall.std() * 2))\n",
    "f = cross_val_score(clf_U2R, X_U2R_test, Y_U2R_test, cv=10, scoring='f1_macro')\n",
    "print(\"F-measure: %0.5f (+/- %0.5f)\" % (f.mean(), f.std() * 2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# RFECV for illustration"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Automatically created module for IPython interactive environment\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYwAAAEbCAYAAADJWrOxAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xl8nHW5///XO1vbpEu6Ubq3lLVAKVjKqiwCsgmCInBE\nFEEO57ApR47gjj89wFH4yVGOnKoIKIKCgAhVENkUkdLSHVoo3RdouqRJk7bJzFzfP+5P0mk6y522\nk0nC9Xw85jEz93rdWeaaz/3ZZGY455xz+ZQUOwDnnHNdgycM55xzsXjCcM45F4snDOecc7F4wnDO\nOReLJwznnHOxeMJwzjkXS1mcjSTtBRwHDAO2APOA6WaWKmBszjnnOhHl6rgn6STgJmAAMBNYC/QE\n9gfGAY8Cd5hZXeFDdc45V0z5EsYPgB+b2fIM68qAs4FSM/t94UJ0zjnXGeRMGM4551yLWJXekq6X\n1FeRX0h6Q9JphQ7OOedc5xG3ldQXQj3FaUB/4LPAbQWLyrk9SNJSSVskbZb0nqT7JPVOW3+fpKaw\nvuVxYYZ9Wx4/Sdt3aPgStUZSvaQFkm6RVBVefyFDPNdLmp4n1npJtZL+IekqSXG/3FVIukPSyhDr\nUkk/av9PzbmdxU0YCs9nAr8ys/lpy5zrCj5uZr2BicDhwM1t1v+3mfVOe/y27b5pj2sAJA0AXgV6\nAceYWR/gVKAfUaOQ+4FLM8Ty2bAuV6x9gNFEX8y+Cvwi5nXeDEwCJgN9gBOBN2Lu61xOcRPGDEnP\nEiWMZyT1AbxJretyzOw94BmixLG7bgDqgUvMbGk4/goz+5KZzQF+BRwvaXTLDpLGAxOAh2LEusnM\nngQuBD4n6ZBwjH6SHpBUI2mZpG+klUCOBB43s9UWWWpmD+yBa3UudsK4nKh57ZFm1ghUAJcVLCrn\nCkTSCOAMYNEeONwpwGPZ+iOZ2UrgBaISRYvPAlPNbF3ck5jZNGAl8OGw6MdEpZh9gBOISjEt/4//\nBG6Q9O+SDpXkdwLcHhMrYYR/iPeB8ZI+AhwMVBcyMOf2sCck1QMriPoTfbvN+q+EOoNaSW0/zJ9I\nW1cr6Yth+UBgTZ7z3k9IGKEU8Bly347KZjUwQFIpcBFws5nVh5LNHWxPSrcCt4fzTAdWSfrcLpzP\nuZ3E7el9O1Gx+E0gGRYb8HKB4nJuT/uEmT0n6QTgN8AgoDZt/Q/N7Bu59s2wfD0wNM95HwP+V9LR\nQGV4PN2+0AEYDmwgirscWJa2bllYj5klgbuBuyX1Ar4A3Ctpmpm9tQvnda5V3FtSnwAOMLMzzezj\n4XFOIQNzrhDM7CXgPuCHe+BwzwHn5WrBFG7hPkp02+izwMNm1tSek0g6kigh/B1YBzQTVYi3GAWs\nynDuLWZ2N7ARGN+eczqXSdyEsZjoW41z3cGPgFMlHbabx7kT6Avc31KxLWm4pDslTUjb7n6iEvon\nacftqND36WzgYeDXZjY3lCB+B3xfUp9w3huAX4d9viTpREm9JJWF21F9iIb2cW63xLolBTQCsyT9\nFdjWstDMritIVM4VkJnVSHoA+BbRh3g+f5SUTHv/FzM7z8w2SDoW+B7wmqQqom/6D7FjpfrLwCZg\nq5m9HvN8CaKWiG8SJaZ70tZfS1TxvRjYCvwMuDesaySq09iX6Lbx28AnzWxxjPM6l1OsoUGyVZqZ\n2a5U3jnnnOuCYo8lJamCaJRagIVm1lywqJxzznU6cYcbOBF4h6j1xf8Cb4fmtbn2uVfSWknzsqyX\npP+RtEjSHElHpK07XdLCsO6m2FfjnHOuYOJWet8BnGZmJ5jZR4CPAf9/nn3uA07Psf4MYL/wuBL4\nKUBoZ353WD8euDj0jnXOOVdEcRNGuZktbHljZm+Tp9WUmb1M1G48m3OBB8LwBf8EqiUNJRoDZ5GZ\nLQ7NDx8O2zrnnCuiuK2kpkv6OaHpHtt7ke6O4US9blusDMsyLT8qzgEHDRpkY8aM2c2wnHPug2PG\njBnrzGxwnG3jJox/A64GWprR/o2oLqPoJF1JdEuLUaNGMX367uYx55z74JC0LP9WkVgJw8y2EbUF\nv3NXg8pgFTAy7f2IsKw8y/JssU0BpgBMmjTJpw90zrkCyZkwJP3OzD4taS5RJ6AdmNmEDLvF9SRw\njaSHiW45bTKzNZJqgP0kjSVKFBcB/7Ib53HOObcH5CthXB+ez27vgSU9RDR5yyBJK4lGBy0HMLN7\ngKlE82ssIuqdellYl5B0DdGcBaXAvWHCJuecc0WUM2GYWcvQzf9uZl9NXxdGsP3qznu17ntxnmMb\nUb1IpnVTiRKKc865TiJus9pTMyw7Y08G4pxzrnPLV4fxb8C/A/tImpO2qg/wSiEDc84517nkq8P4\nDfAnolm80ofoqDezXJ3ynHPOdTP56jA2EQ3LfDGApL2AnkBvSb3NbHnhQ+zcXly4lrfW1FNdWU51\nr3L6VZZT3auC+q3NLF3fwLrNTRy4dx8mjqxmYO8exQ7XuW5tWyLJ42+s4kOj+7PfkD67fbz6rc3M\nX13H6IGVDO3Xaw9E2LXFnaL140R9MIYRzYc8GniLaG7vD6yX3q7hsvteJ+aAvxwxqprrT9mfj+w3\nCEkFjS2VMv65ZD1PzlpN/bYEFx05kuP3jc6bTBkpM8pLc1dhNSVSAFSU5d7OzEgZlJbs+WtKpaIf\nbskeOvbKjY0AjOhfuVvHqW1s4uV31vHiwrXUb01w5qF7c9r4vanqseO/1LZEkhIp78+6vRLJFImU\n0bO8dKd1ZsbidQ1U9yov6JeUZMowM8raeW31W5u556V3WfjeZo4dN5ATDxjMPoN7t67f2NDEA68u\nY0T/XnzskL3p3SP/x5SZ8cz89/mvqW+xfEMjfXqW8cvPH8mkMQMA2NqcZN3mbdQ2NpNMGfsP6UOv\nip1/dgAbGpp4aNpynpy1mrfX1mMGPctL+I9TD+Cy48awYuMWps5dw3579ea0g/du3W9t/VZWbdxC\ndWUFA3tX0LfnzqMnJVPGXc+9zfzVdUwYUc2Ekf3Yd3BvhlX3QsDa+m0s39DIhoYo1p7lpUwY0Y+x\ng6paPzPqtzbzyqL1TF+6gXF79ebEAwZ3WDKLOx/GbOBk4DkzO1zSScAlZnZ5oQNsj0mTJlkhe3q/\n/X49FaUljBlUxcqNjZz947+zd9+e/OaLR7MtkWRjQzO1W5rY1NhMZY8yxg6son9VOW+tqWf6sg08\n+M/lrKrdwsHD+rL/kD7061XOQUP7cPohQ+nXq30TGtZtbWbD5iaqK8upKCvhtSUbeGlhDQvfq6d2\nSzPv121lQ0MTVRWl9CgvZUNDE2MGVlJSIlZsaKS8tITTxg/h3InDGVYd/bGtqm3kxYU1/H3ROt7f\ntJWGpiRVFaV8/rgxXHH8PtRtbeYPs1bzztrNHDysL+OH9mXOylqemLWa5RsaOfewYXzu2DEM7tOD\nJesaWLa+gSXrGlm+oYFe5WWMGVjJsOpelJUKM2hoSlDb2My2RIoR1b0YM6iKvfv2pLqqnPqtCX79\nz2U8PG05Bnxkv8FMGtOfmvptLFnXwOZtCQBKJPr2LKO6sqK1lFdRVspba+qYvbKWrc1JRg+sYlDv\nCqYt2cC7NQ0AHD6qmrMnDGNQ7woAKkpL6FdZTo+yEqYt2chLb69lWyLFWYcO5cxDh7J+cxOzVtYy\na3kts1ZsbD1OdWU5leWlrN60lV7lpZw6fgifOHwYg3v35IFXl/KH2atpSqTo3aOMqh6liOiffq++\nPRg9sIqR/XsxoKqCfr3K6d9yDZXl9OsVLdvY2MSSdQ28s3Yzs1fUMmdlLWtqt1K/LUFFaQmfPnIE\n/37ivvTtVc4/Fq3jxbdreGlhDatqt1BRWsLZE4Zy4ZEjqa6swDCWrW9k1opa3nm/nkRIxj3LSqmu\nLKdvr3JKwofSxJH9+NjBe+/wxWZbIsnrSzbyt0U1zFxey9yVm5Dg2HGDOH7fgUiitrGZ/lXlnHHI\nUAb36UHDtgR/efP91gTWnEzxs78tYd3mbYzo34uVG7e0/j4+f+wYAL77xzdZ3xDNYtuzvIQPje6/\nU8Kt6lHGocP7cfCwvsxcXssTs1axuKaB/Yf05uqT9uWu595h9aYt3PixA5m9opa/vPk+W5q3z4FV\nWiIO3LsPowdWUl1ZQa/yUuq2NLO+oYm/L1pHUyLFUWMHcOy4QYwf1pffvr6c595ay6DePVi3uXUO\nOa49eV++fMr+PDZzFd/+wzwamrafY0T/XkwcWc1x+w7ijEP2pqy0hOsemsnzC9YyakAlKzY2tn7Z\nrCgtoaQEtjanMv6/9+lR1vplZN3mbSRSRlmJWn+HBw/ryx+uPq7dyRtA0gwzmxRr25gJY7qZTQqJ\n43AzS0mabWa7O8XlHlXIhNGcTDHpe8+xaUszJx4wmLV121ixsZE/XnM8YwZVxTpGUyLF799Yye+m\nr2D95iY2NjS1/uMfM24gJYLaLc1RUhpYxd79erK2fitL1jWQMhgzsJLBfXrw+tKNvLFsY+sfS4ue\n5SWMH9qXAVU9GFBVzof3G8wpBw1BgqfnrOGJWauoqihj9KBK6rY0M3Xue2za0rzTMY7ZZyBjB/Wm\nf2U5C96vZ+rcNZSXlrSWOPbu25P36ra27jN5zABGD6zkqTlrdvinBCgvFSP7V9LYlNxhn3QSGUtp\nJYJTxw+hd49yXnq7hnWbt1GiqHTQvzJKsEkz6rYk2NjYRP3WROu+vXuUMWFEP6p6lLFsfQPv121j\nwoh+nHjAXjQnUzwxcxUL3qvP+rs6cO8+lJWKeavqdlg+qHcFE0dWM3FkNcfuO4jDRlQjYMbyjTwx\ncxVPz11DbWP0M+1VXsp5Rwxn7749qW1spiEkuZQZ79VtZen6BlbXbiWZildEHVAVnXvUgEr6V1aw\nZtMWfv/Gytb1zUmjqqKU4/YdxEf2H8w779fz6IyVO3yIAZSViH336k2PshKM6Nt3bWMzdVubSVn0\nbb05aZx84F586+zxLHivnidnr+LFhTU0NiUpLxUHD+vHxJHVNCdTvBgSVLrSEjFxZDVvrq7b6W9i\n0uj+fOvj45kwopoVGxp5Zv57PPjacpasi5LwYSP6cev5E9jSnOCJmauZs2rTTn8gGxqbWLFh+zmP\nGjuAT35oBOcfPpyy0hLWbd7G5+6dxvzVdVRXlnPWoUM5bEQ1/SrLMYO5q2qZvWITazZtYdOWZhqb\nkvTtGSXrD43uz+ePHbPDLS0z4+m5a3jsjVUcvc8AzjhkKD95fhG/nb6CfQZVsXhdA5PHDuCLH96H\n+q3NrK3fxtyVm5i5fCOrN22lvFQM6t2DtfXbuOWcg7nk6NHUb21m3qo6lq5vYOm6BhIpY8ygKkYP\nqGRQ7x5UV5ZTt7WZWctrmb+6rvX/b0DvCk7YfzBHjOrP0vUNvLhwLe/XbeObZ+/aoN6FSBjPAZ8g\nqvweRHRb6kgzO3aXIiyQQiaMV99dz8U/+ydnHTqUaUs3UFO/jSmf/dAORdL2MjPmrtrEEzNX87d3\nauhZHn3T29KUZOn6RtZt3sbAqgrGDKqiRLB0fSM19dsYP7QvJx04mH0G9WbTluiDaMLIao4aOyDj\nLYpstiWSvLZ4Q+u39epe5Rwxuv9Ox3j7/Xp+9eoyhvfvxTmHDWNYdS82NDQxf/Umxg6qar29s6mx\nmSfnrMbMGD2wirEDqxhW3bP1W09jU4L367bR8jdXWVFGdWU5ZSViVe0WlqyL6nxqG5tIpoyzJgxt\nPXYqFX3IDurdI+stskQyRd3WBI1NCYb265X3Ftnq2i1sDR9mW5tT1G5pomFbkkOH92Pvfj0BWLR2\nMy8sWMve/XoycWQ1I/r3ynk7sSmR4qW3a6ip38ZZE/KXHM2M+m0JakPptLaxmY2NTWza0kxtYzPV\nleWMHljFPoOqMp57Ve0Wfvn3JZSWiBMOGMyk0QN2+PnUb23mH++ub01KQ/r25OBhfXP+nSRTxn3/\nWMoPn1nY+mE/qHcFpx+yNycdsBfHjBtIZcX220RmxppNW6koK6Ffr3KWrmvgiVmreOntGiaMqOa8\nw4dz+MhqNm9L0NCUZFi/njtdRyplvPRODXVbmjl7wrBYtzfXb97Gm2vq2Gdwb4ZX73xLpmFbgtkr\na3f6mewpZsZPX3qXH/91EdecvC9XnTBup7jNjPmr63hi5ipmr6zl6pP25cQD9trjseyOQiSMKmAL\nUb+NzwD9gAfNbP3uBLqnFTJh3Dr1Le59ZQmzvnUa5aUlvLdpK6MG7t598Hyak6mdiuKJZGqXip3O\ntdeKDY08OmMlR4zuz3HjBvrfXRaplO2xOrZiaE/CiDta7V7AGjPbCtwvqRcwBOhUCaOQnl+wlqPG\nDmy9j1joZAFkrCj1f1rXUUYOqOTLp+6ff8MPuK6cLNor7qfPI0B6bUwyLPtAWLGhkXfWbuakAztX\nUdI55zpS3IRRFma/AyC8rihMSJ3PiwvXAnCyJwzn3AdY3IRRI+mcljeSzgXWFSakzuf5BWsZM7CS\nsTFbQznnXHcUtw7jKuBBST8BRDSF6qUFi6oT2dKU5B/vrudfjhpV7FCcc66o4s649y5wtKTe4f3m\ngkbVibR04DqpkzWFc865jpZvtNpLzOzXkm5osxwAM9uTU7Z2Oqtqt/CNJ+YxdlAVR+0zoNjhOOdc\nUeUrYbS0Hd39Uby6mMamBF+8fzrbmlM8fOUkepTF7xDnnHPdUb6EMS48v2lmH5hmtGbGjY/O4a33\n6rj380ey71698+/knHPdXL5WUmcquv90c0cE01k8PXcNT89Zw1dOO8DrLpxzLshXwvgzsJFo/ov0\nUdhENC1334JFViSbGpv5zpNvcujwfvzrR/YpdjjOOddp5CxhmNmNZlYNPG1mfdMefbpjsgC49U9v\nsbGxiVvPP9SH4XDOuTSxPhHN7NxCB9IZzFi2gYdfX8EVx4/lkOH9ih2Oc851KjkThqS/h+d6SXXh\nueVRl2vfsN/pkhZKWiTppgzr+0t6XNIcSdMkHZK27npJ8yTNl/SlXbm49npjWS0AV50wLs+Wzjn3\nwZPvltTx4blP2q2oPnFuSUkqBe4GzgDGAxdLajvDx9eAWWY2gajn+F1h30OALwKTgcOAsyXt2/7L\na5+WCYmyTd3onHMfZLFuSUkaJ6lHeH2ipOskVefZbTKwyMwWh8EKHwba3toaDzwPYGYLgDGShgAH\nAa+ZWaOZJYCXgPNjX9UuSqaiAXlLCjzftnPOdUVxa3V/DyTDt/wpwEjgN3n2GU405lSLlWFZutmE\nRCBpMjAaGAHMAz4saaCkSuDMcM6CSoYB3Ms+QOPbO+dcXHETRip80z8P+LGZ3QgM3QPnvw2oljQL\nuBaYCSTN7C3gduBZoqa9s4jm4NiJpCslTZc0vaamZreCaS1heMJwzrmdxE0YzZIuBj4HPBWW5Z6s\nGFaxY6lgRFjWyszqzOwyM5tIVIcxGFgc1v3CzD5kZh8h6gvydqaTmNkUM5tkZpMGDx4c83IyS5p5\n6cI557KImzAuA44Bvm9mSySNBX6VZ5/Xgf0kjZVUAVwEPJm+gaTqsA7gCuBlM6sL6/YKz6OIblvl\nuwW22xIpizX5vHPOfRDFHd78TeA6iJrCAn3M7PY8+yQkXQM8A5QC95rZfElXhfX3EFVu3y/JgPnA\n5WmH+L2kgUAzcLWZ1bbv0tov5QnDOeeyipUwJL0InBO2nwGslfSKmd2Qaz8zmwpMbbPsnrTXrwIZ\nZ5k3sw/HiW1P8hKGc85lF/eWVL9wq+h84AEzOwo4pXBhFUfSE4ZzzmUVN2GUSRoKfJrtld7dTjLl\nld7OOZdN3ITxXaK6iEVm9rqkfYB3ChdWcXgJwznnsotb6f0I8Eja+8XAJwsVVLEkU0ap9/J2zrmM\n4lZ69yRqwXQw0LNluZl9oUBxFUUyZZSWesJwzrlM4t6S+hWwN/AxonGdRgD1hQqqWBJewnDOuazi\nJox9zeybQIOZ3Q+cBRxVuLCKI2leh+Gcc9nEHhokPNeGocf7Ad1usutk0hOGc85lE6sOA5gSenh/\nk2h4j97AtwoWVZFEJQyfltU55zKJ20rq5+HlS8A+hQunuLwfhnPOZZczYUjKN/THnXs2nOJKpsyH\nNnfOuSzylTD6dEgUnYSXMJxzLrucCcPMbumoQDqDRCrlzWqdcy6LuHN6358+h7ek/pLuLVxYxZFK\n4a2knHMui7hNgiakz0dhZhuBwwsTUvEkUinKvKe3c85lFDdhlIRmtQBIGkD8JrldRtKgxG9JOedc\nRnE/9O8AXpXUMgDhBcD3CxNS8SRTKa/0ds65LOL2w3hA0nTg5LDo/DBta7eSSHqzWuecyyb2baWQ\nILpdkkiXMm9W65xz2fg4GGkS3nHPOeey8oSRJuUd95xzLqu4/TBuj7Osq0v4FK3OOZdV3BLGqRmW\nnZFvJ0mnS1ooaZGkmzKs7y/pcUlzJE0LQ6e3rPuypPmS5kl6KMz6V1Apn0DJOeeyypkwJP2bpLnA\ngeFDveWxBJibZ99S4G6ixDIeuFjS+DabfQ2YZWYTgEuBu8K+w4HrgElmdghQClzU/strn0TKvOOe\nc85lka+V1G+APwG3AuklhHoz25Bn38nAIjNbDCDpYeBcdmxpNR64DcDMFkgaI2lIWmy9JDUDlcDq\nGNezW5Ip8457zjmXRc4ShpltMrOlRN/8N5jZMjNbBiQk5ZuidTiwIu39yrAs3WzgfABJk4HRwAgz\nWwX8EFgOrAE2mdmzmU4i6UpJ0yVNr6mpyRNSbklvVuucc1nFrcP4KbA57f3msGx33QZUS5oFXAvM\nBJJhGJJzgbHAMKBK0iWZDmBmU8xskplNGjx48G4Fk/SOe845l1XcjnsyM2t5Y2YpSfn2XQWMTHs/\nIixrZWZ1wGUAkgQsARYDHwOWmFlNWPcYcCzw65jx7hIvYTjnXHZxSxiLJV0nqTw8rif6YM/ldWA/\nSWMlVRBVWj+ZvoGk6rAO4Arg5ZBElgNHS6oMieSjwFtxL2pXRc1qvWuKc85lEvfT8Sqib/iriOoi\njgKuzLWDmSWAa4BniD7sf2dm8yVdJemqsNlBwDxJC4laU10f9n0NeBR4g6g1VgkwpR3XtUuSKaPU\n84VzzmUUd/DBtexCs1YzmwpMbbPsnrTXrwL7Z9n328C323vOXWVmIWF4xnDOuUzi9vTeX9JfJc0L\n7ydI+kZhQ+tYqVBD4x33nHMus7hfp38G3Aw0A5jZHDqgI11HSoaM4R33nHMus7gJo9LMprVZltjT\nwRRTS8LwsaSccy6zuAljnaRxgAFI+hRRh7puIxlaDfstKeecyyxuP4yriVopHShpFVF/ic8ULKoi\nSCa9hOGcc7nkTRiSSogGATxFUhVQYmb1hQ+tYyVSKcAThnPOZZP3lpSZpYD/DK8bumOygLRbUp4w\nnHMuo7h1GM9J+oqkkZIGtDwKGlkH80pv55zLLW4dxoXh+eq0ZQbss2fDKR5PGM45l1vcOoxLzOyV\nDoinaFr7YXjCcM65jOLWYfykA2IpqoSXMJxzLqe4dRh/lfTJMHJst5TyhOGccznFTRj/CjwCNEmq\nk1Qvqa6AcXW41hJG982Jzjm3W+KOVtun0IEUm1d6O+dcbnFbSSHpHOAj4e2LZvZUYUIqDh980Dnn\ncos7vPltRJMbvRke10u6tZCBdbSWjnslfkvKOecyilvCOBOYGFpMIel+YCbRkOfdwvZmtT6BknPO\nZdKeT8fqtNf99nQgxZYIgw96vnDOucziljBuBWZKegEQUV3GTQWLqghS5iUM55zLJW4rqYckvQgc\nGRZ91czeK1hURbC9416RA3HOuU4qbqX3eUCjmT1pZk8CWyV9orChdaztHfc8YzjnXCZxPx2/bWab\nWt6YWS3w7cKEVBwJH0vKOedyipswMm0XZ+DC0yUtlLRI0k51HpL6S3pc0hxJ0yQdEpYfIGlW2qNO\n0pdixrpLWlpJebNa55zLLG7CmC7pTknjwuNOYEauHSSVAncDZwDjgYsljW+z2deAWWY2AbgUuAvA\nzBaa2UQzmwh8CGgEHo99VbvAO+4551xucRPGtUAT8FvgYWArO86NkclkYJGZLTazprDfuW22GQ88\nD2BmC4Axkoa02eajwLtmtixmrLukZYpWL2E451xmcVtJNdD+ZrTDgRVp71cCR7XZZjZwPvA3SZOB\n0cAI4P20bS4CHsp2EklXAlcCjBo1qp0hbre9Wa0nDOecy6TYTYJuA6olzSIqxcwEki0rJVUA5xCN\nlJuRmU0xs0lmNmnw4MG7HEhLxz0ffNA55zKLPfjgLlgFjEx7PyIsa2VmdcBlAGGujSXA4rRNzgDe\nMLP0EkdBtJQwPGE451xmOUsYkm4PzxfswrFfB/aTNDaUFC4Cnmxz/OqwDuAK4OWQRFpcTI7bUXuS\nN6t1zrnc8t2SOjN882/3IINmlgCuAZ4B3gJ+Z2bzJV0l6aqw2UHAPEkLiUoT17fsL6kKOBV4rL3n\n3hWtzWo9YTjnXEb5bkn9GdgI9A4z7Amwlmcz65trZzObCkxts+yetNevAvtn2bcBGJjvAvaUpJcw\nnHMup5wlDDO70cyqgafNrK+Z9Ul/7qAYO4SXMJxzLre4zWrPDf0jWgYffM3MagoXVsfzEoZzzuUW\nd/DBC4BpwAXAp4Fpkj5VyMA6WsLn9HbOuZziNqv9BnCkma0FkDQYeA54tFCBdbTW0Wq9p7dzzmUU\ne/DBlmQRrG/Hvl2ClzCccy63uCWMP0t6hu19Ii6kTeunri6ZMkoE8hKGc85lFLfS+0ZJ5wPHh0VT\nzKygo8d2tKSZT8/qnHM5xB4axMweo4M60RVDMmV4vnDOuez8IzJIpryE4ZxzufgnZJBMmVd4O+dc\nDrFvSYVBAg8kGhpkYZgUqdtIpFKeMJxzLodYCUPSWcA9wLtE40iNlfSvZvanQgbXkZIpb1LrnHO5\nxC1h3AGcZGaLACSNA54GulHCSHmnPeecyyFuHUZ9S7IIFgP1BYinaLyE4ZxzueUsYYS+FwDTJU0F\nfkdUh3EB0QRJ3UYylaKs1BOGc85lk++W1MfTXr8PnBBe1wC9ChJRkSTNx5FyzrlcciYMM7usowIp\ntqS3knKgbO0vAAAYqElEQVTOuZzitpIaDHwRGJO+j5l9oTBhdbxE0vthOOdcLnFbSf0B+BvRkObJ\nwoVTPCnzhOGcc7nETRiVZvbVgkZSZAnv6e2ccznFbVb7lKQzCxpJkfnQIM45l1vchHE9UdLYIqlO\nUr2kunw7STpd0kJJiyTdlGF9f0mPS5ojaZqkQ9LWVUt6VNICSW9JOib+ZbVfNPigJwznnMsm7nwY\nfdp7YEmlwN3AqcBK4HVJT5rZm2mbfQ2YZWbnSTowbP/RsO4u4M9m9qkwjlVle2Noj0TKKPFmtc45\nl1XOEoakMXnWS9KILKsnA4vMbHEYqPBh4Nw224wHngcwswXAGElDJPUDPgL8IqxrMrPaPNeyW1Ip\n8457zjmXQ75bUj+Q9HtJl0o6WNJekkZJOlnS/we8AhyUZd/hwIq09yvDsnSzgfMBJE0GRgMjgLFE\nnQN/KWmmpJ9LqmrfpbWPlzCccy63nAnDzC4AvgkcQHS76G9ETWyvABYCJ5vZX3bj/LcB1ZJmAdcC\nM4ma7ZYBRwA/NbPDgQZgpzoQAElXSpouaXpNTc0uB5Iyr8Nwzrlc8tZhhDqHr+/CsVcBI9PejwjL\n0o9dB1wG0e0tYAnRwIaVwEozey1s+ihZEoaZTQGmAEyaNMl2IU7AO+4551w+hZxx73VgP0ljQ6X1\nRcCT6RuEllAV4e0VwMtmVmdm7wErJB0Q1n0USK8s3+O8455zzuUWe8a99jKzhKRrgGeAUuBeM5sv\n6aqw/h6i+o/7JRkwH7g87RDXAg+GhLKYUBIplITP6e2cczkVLGEAmNlUYGqbZfekvX4V2D/LvrOA\nSYWML10yZZR4CcM557KK9ZU6NJ+9RNK3wvtRoVVTt+Ed95xzLre492D+FzgGuDi8rydqNdVtJL1Z\nrXPO5RT3ltRRZnaEpJkAZrYxrbK6W/AShnPO5Ra3hNEchvowaJ0fI1WwqIogkTJKvae3c85lFTdh\n/A/wOLCXpO8Dfwf+q2BRFUEylfIpWp1zLoe4gw8+KGkGUX8IAZ8ws7cKGlkH8+HNnXMut7wJI9yK\nmm9mBwILCh9ScXjCcM653PLekjKzJLBQ0qgOiKdokj6WlHPO5RS3lVR/YL6kaUQDAQJgZucUJKoi\n8I57zjmXW9yE8c2CRtEJeLNa55zLLW6l90uShgBHhkXTzGxt4cLqWKmUkTK8DsM553KIOzTIp4Fp\nwAXAp4HXJH2qkIF1pKRFo6J7s1rnnMsu7i2prwNHtpQqQse954jmqejykqmQMLzjnnPOZRW3415J\nm1tQ69uxb6fXmjC8hOGcc1nFLWH8WdIzwEPh/YXAnwoTUsdrvSXldRjOOZdV3ErvGyWdDxwfFk0x\ns8cLF1bHSiajhOGtpJxzLrtYCUPSWGCqmT0W3veSNMbMlhYyuI7iJQznnMsvbj3EI+w4Om0yLOsW\nWuswfIpW55zLKu4nZJmZNbW8Ca+7zXwYidaEUeRAnHOuE4v7EVkjqXUYEEnnAusKE1LHS3kJwznn\n8orbSuoq4EFJPyEa3nwFcGnBoupgXsJwzrn84raSehc4WlLv8H5zQaPqYF6H4Zxz+cUdGuR6SX2J\nRqr9kaQ3JJ0WY7/TJS2UtEjSTRnW95f0uKQ5kqZJOiRt3VJJcyXNkjS9PRfVXi0Jw5vVOudcdnG/\nUn/BzOqA04CBwGeB23LtECZeuhs4AxgPXCxpfJvNvgbMMrMJRLe47mqz/iQzm2hmk2LGuUsSqagB\nWIn39HbOuaziJoyWT9IzgQfMbH7asmwmA4vMbHFoVfUwcG6bbcYDzwOY2QJgTBgVt0OFfOElDOec\nyyFuwpgh6VmihPGMpD7s2C8jk+FEleMtVoZl6WYD5wNImgyMBkaEdQY8J2mGpCtjxrlLWkoY3nHP\nOeeyi9tK6nJgIrDYzBolDQQu2wPnvw24S9IsYC4wk6hTIMDxZrZK0l7AXyQtMLOX2x4gJJMrAUaN\n2rVZZFPe09s55/KK20oqBbyR9n490Yi1uawCRqa9HxGWpR+3jpB4JAlYAiwO61aF57WSHie6xbVT\nwjCzKcAUgEmTJlmc62krkfSE4Zxz+RSyHenrwH6SxkqqAC4CnkzfQFJ1WAdwBfCymdVJqgq3vZBU\nRVTZPq9QgfpYUs45l1/cW1LtZmYJSdcAzwClwL1mNl/SVWH9PcBBwP2SDJhPdOsLYAjweFTooAz4\njZn9uVCxerNa55zLL3bCCM1kh6TvY2bLc+1jZlOBqW2W3ZP2+lVg/wz7LQYOixvb7mrp6V3iCcM5\n57KKO7z5tcC3gffZ3jrKgAkFiqtDpbyE4ZxzecUtYVwPHBAqu7ud1hKGd9xzzrms4lZ6rwA2FTKQ\nYmotYZR6wnDOuWziljAWAy9KehrY1rLQzO4sSFQdLOG3pJxzLq+4CWN5eFTQjSZOapH0W1LOOZdX\n3I57twB09+HNy3x4c+ecyyru8OaHSJpJ1Fdifhjf6eDChtZxWksYni+ccy6ruB+RU4AbzGy0mY0G\n/gP4WeHC6lgtPb29hOGcc9nF/YSsMrMXWt6Y2YtAVUEiKoKElzCccy6v2K2kJH0T+FV4fwlhkMDu\nIOV1GM45l1fsGfeAwcBj4TE4LOsWEikffNA55/KJ20pqI3BdgWMpmqRPoOScc3nlTBiSfmRmX5L0\nR6Kxo3ZgZucULLIOlPQpWp1zLq98JYyWOosfFjqQYmopYXjHPeecyy5nwjCzGeHlRDO7K32dpOuB\nlwoVWEfyEoZzzuUXt9L7cxmWfX4PxlFUyVQKyefDcM65XPLVYVwM/AswVlL69Kp9gA2FDKwjJVJG\nqd+Ocs65nPLVYfwDWAMMAu5IW14PzClUUB0taeYtpJxzLo98dRjLgGXAMR0TTnEkk54wnHMun7iD\nDx4t6XVJmyU1SUpKqit0cB3FSxjOOZdf3ErvnwAXA+8AvYArgLsLFVRHS6Y8YTjnXD6xB08ys0VA\nqZklzeyXwOmFC6tjJVPmTWqdcy6PuAmjUVIFMEvSf0v6cpx9JZ0uaaGkRZJuyrC+v6THJc2RNE3S\nIW3Wl0qaKempmHHuEi9hOOdcfnETxmeBUuAaoAEYCXwy1w6SSoluW50BjAculjS+zWZfA2aZ2QTg\nUuCuNuuvB96KGeMu82a1zjmXX6yEYWbLzGyLmdWZ2S1mdkO4RZXLZGCRmS02sybgYeDcNtuMB54P\n51gAjJE0BEDSCOAs4OftuJ5dkkoZpaWeMJxzLpd8HffmkmHQwRahZJDNcGBF2vuVwFFttpkNnA/8\nTdJkYDQwAngf+BHwn0SdBAvKSxjOOZdfvo57Z4fnq8Nz+gRKWRNJO9wG3CVpFjAXmAkkJZ0NrDWz\nGZJOzHUASVcCVwKMGjVql4LwZrXOOZdfnI57SDrVzA5PW/VVSW8AO1Vkp1lFVNfRYkRYln78OuCy\ncA4BS4hm8rsQOEfSmUBPoK+kX5vZJRlinEI05ziTJk3apSTmHfeccy6/uJXeknRc2ptjY+z7OrCf\npLGhhdVFQPp4VEiqDusg6tvxcqgnudnMRpjZmLDf85mSxZ6SSBmlPj2rc87lFHdO78uBeyX1AwRs\nJM8UrWaWkHQN8AxRC6t7zWy+pKvC+nuAg4D7JRkwP5ynw6XM+2E451w+cadonQEcFhIGZrYp5n5T\ngaltlt2T9vpVYP88x3gReDHO+XZVImU+tLlzzuWRr5XUJWb2a0k3tFkOgJndWcDYOkzKe3o751xe\n+UoYVeG54E1biymRSnmzWuecyyNfK6n/C8+3dEw4xZFK4a2knHMuj3y3pP4n13ozu27PhlMciVSK\nHuVx6/+dc+6DKd+n5IwOiaLIkimjxG9JOedcTvluSd3fUYEUU9Kb1TrnXF6x7sNIGgx8lWiwwJ4t\ny83s5ALF1aESSW9W65xz+cTt3vwg0TDjY4FbgKVEPbm7Be+455xz+cVNGAPN7BdAs5m9ZGZfALpF\n6QK8455zzsURt2lQc3heI+ksYDUwoDAhdTzvuOecc/nFTRjfC8OC/AfwY6Av8OWCRdXBEj5Fq3PO\n5RU3YbwWxo/aBJxUwHiKIukTKDnnXF5x6zBekfSspMsl9S9oREWQTBllPkWrc87lFHdO7/2BbwAH\nAzMkPSWpYPNTdDTvuOecc/nFnjXIzKaZ2Q3AZGAD0G069XnHPeecyy9WwpDUV9LnJP0J+Aewhihx\ndAtJ77jnnHN5xa30ng08AXw3THrUrSS8Wa1zzuUVN2HsY2ZW0EiK6GMHD+GgoX2LHYZzznVqcado\n7bbJAuBHFx1e7BCcc67Ti13p7Zxz7oPNE4ZzzrlY4raS+u/QUqpc0l8l1XSnfhjOOefyi1vCOM3M\n6oCziYY23xe4Md9Okk6XtFDSIkk3ZVjfX9LjkuZImibpkLC8Z3g/W9J8Sd16TnHnnOsK4iaMlsrx\ns4BHwrhSOUkqBe4GziCaeOliSePbbPY1YJaZTQAuBe4Ky7cBJ5vZYcBE4HRJR8eM1TnnXAHETRhP\nSVoAfAj4a5iBb2uefSYDi8xssZk1AQ8D57bZZjzwPICZLQDGSBpikc1hm/Lw6NYttZxzrrOLO5bU\nTcCxwCQzawYa2PnDv63hwIq09yvDsnSzgfMBJE0GRgMjwvtSSbOAtcBfzOy1OLE655wrjLiV3hcQ\nzbaXlPQN4NfAsD1w/tuA6pAYrgVmAkkAM0ua2USiBDK5pX4jQ2xXSpouaXpNTc0eCMk551wmcXt6\nf9PMHpF0PHAK8APgp8BROfZZBYxMez8iLGsVKtIvA5AkYAmwuM02tZJeAE4H5rU9iZlNAaaEY9RI\nWhbzmgAGAevasX1n1l2uxa+j8+ku1+LXkdnouBvGTRjJ8HwWMMXMnpb0vTz7vA7sJ2ksUaK4CPiX\n9A0kVQONoY7jCuBlM6sLdSTNIVn0Ak4Fbs8XpJkNjnk9LeefbmaT2rNPZ9VdrsWvo/PpLtfi17H7\n4iaMVZL+j/DBLakHeW5nmVlC0jXAM0ApcK+ZzZd0VVh/D3AQcL8kA+YDl4fdh4blpeE8vzOzp9p5\nbc455/aguAnj00S3hH4YvvUPJUY/DDObCkxts+yetNevAvtn2G8O4AM8OedcJxK3lVQj8C7wsVBq\n2MvMni1oZB1jSrED2IO6y7X4dXQ+3eVa/Dp2k+IMRCvpeuCLwGNh0XlEdRk/LmBszjnnOpG4CWMO\ncIyZNYT3VcCroYe2c865D4C4Pb3F9pZShNddeoq6fONcdVaSRkp6QdKbYZyt68PyAZL+Iumd8Ny/\n2LHGETpozpT0VHjfVa+jWtKjkhZIekvSMV3xWiR9OfxdzZP0UBjXrdNfh6R7Ja2VNC9tWda4Jd0c\n/vcXSvpYcaLOLMu1/CD8bc0J4+9Vp63rsGuJmzB+Cbwm6TuSvgP8E/hFwaIqsJjjXHVWCeA/zGw8\ncDRwdYj9JuCvZrYf8Nfwviu4Hngr7X1XvY67gD+b2YHAYUTX1KWuRdJw4DqiER0OIWrdeBFd4zru\nI2qYky5j3OH/5SLg4LDP/4bPhM7iPna+lr8Ah4S7Om8DN0MRrsXMYj2AI4j+mK4DDo+7X2d8AMcA\nz6S9vxm4udhx7eK1/IGoufNCYGhYNhRYWOzYYsQ+gugf+WTgqbCsK15HP6JOp2qzvEtdC9uH8xlA\n1ILyKeC0rnIdwBhgXr6ff9v/d6Km/8cUO/5c19Jm3XnAg8W4lrzNakO2mm/RN6c38m3fRWQa5ypX\nr/VOSdIYoubHrwFDzGxNWPUeMKRIYbXHj4D/BPqkLeuK1zEWqAF+KekwYAZRyalLXYuZrZL0Q2A5\nsAV41syeDQOCdpnrSJMt7uFEd0laZBrnrjP7AvDb8LpDryXvLSkzSwILJY0qVBCu/ST1Bn4PfMmi\nIVZaWfRVo1OP7ivpbGCtmc3Itk1XuI6gjKgE/lMzO5xocM4dbtt0hWsJ9/jPJUqAw4AqtZkorStc\nRyZdNe62JH2d6Lb0g8U4f9yOe/2B+ZKmEf0zAGBm5xQkqsLLO85VZyapnChZPGhmLU2d35c01MzW\nhI6Va4sXYSzHAedIOhPoCfSV9Gu63nVA9K1upW0fUflRooTR1a7lFGCJmdUASHqMaJTqrnYdLbLF\n3SX//yV9nmgSu4+GBAgdfC1xK72/SRTod4E70h5dVes4V5IqiCqNnixyTLFIElGDg7fM7M60VU8C\nnwuvP0dUt9FpmdnNZjbCzMYQ/fyfN7NL6GLXAWBm7wErJB0QFn0UeJOudy3LgaMlVYa/s48SVd53\ntetokS3uJ4GLJPVQNNbdfsC0IsQXm6TTiW7fnmNRR+oWHXsteSpe9gWOy7D8eGBcsSuGducBnEnU\n2uBd4OvFjqcdcR9PVLSeA8wKjzOBgUQVyO8AzwEDih1rO67pRLZXenfJ6yCaGXJ6+L08QVQq73LX\nAtwCLCAaGfpXQI+ucB3AQ8AaoJmoxHd5rriBr4f//YXAGcWOP8a1LCKqd235n7+nGNeSs+NeaBt/\ns5nNbbP8UOC/zOzjWXd2zjnXreS7JTWkbbIACMvGFCQi55xznVK+hFGdY12vPRmIc865zi1fwpgu\n6YttF0q6gqituXPOuQ+IfHUYQ4DHgSa2J4hJQAVwnkWtQ5xzzn0AxB2t9iTgkPB2vpk9X9ConHPO\ndTpxJ1B6wcx+HB6eLFxWkkzSHWnvvxIGrNwTx75P0qf2xLHynOeCMOLsCxnW/SCM5vqDXTjuxNBR\nsdOStHkX9/vErgzguavnc8URt+Oec3FtA86XNKjYgaSTFHdUA4javX/RzE7KsO5KYIKZ5Z2iOIOJ\nRH1mYlOkK/yffoJo5GfXjXWFP0TXtSSIppD8ctsVbUsILd8uJZ0o6SVJf5C0WNJtkj4jaZqkuZLG\npR3mFEnTJb0dxqNqmVPjB5JeD/MF/Gvacf8m6Umintdt47k4HH+epNvDsm8RdY78RdtSRDhOb2CG\npAslDZb0+3De1yUdF7abLOlVRfN8/EPSAWFEge8CF0qaFfb/jqSvpB1/nqQx4bFQ0gNEHehGSjot\nHPMNSY+EscQIP6s3w3X/MMM1nhDONyvE0ycsvzHt53VLpl9ktm0kXRqWzZb0K0nHAucAPwjnGRce\nf5Y0I/wODgz7jg3XMVfS9zKd13Vixe7V6I/u9QA2A32BpUTDfn8F+E5Ydx/wqfRtw/OJQC3RENQ9\niMbCuSWsux74Udr+fyb6orMfUS/YnkTf+r8RtulB1ON6bDhuAzA2Q5zDiIbCGEw0ptrzwCfCuheJ\n5oTIeH1pr38DHB9ejyIaroVw/WXh9SnA78PrzwM/Sdv/O8BX0t7PI+rfNAZIAUeH5YOAl4Gq8P6r\nwLeIejIvZHtdZHWGeP9IGK2BKNmVEQ1ZPoVoErQSomHMP9Lmd5JxG6J5F94GBoXtBmT53f4V2C+8\nPopo6BeIhrK4NLy+Ov3n6Y/O/2hPMd25WMysLnw7vo5omOw4XrcwFLWkd4Fnw/K5QPqtod+ZWQp4\nR9Ji4ECiD7cJaaWXfkQJpQmYZmZLMpzvSOBF2z7Q3oNEH4hPxIwXomQwXmqdfLJv+ObfD7hf0n5E\nw7iUt+OYLZaZWcuw1UcT3e55JZyrAngV2ARsJSoNPUX0od7WK8Cd4foeM7OVkk4j+pnNDNv0Jvp5\nvZy2X7ZtDgMeMbN1AGa2oe0Jw8/gWOCRtJ9Nj/B8HPDJ8PpXwO15fxKu0/CE4QrlR0Tzp/wybVmC\ncBs03JevSFu3Le11Ku19ih3/Tts26zOib8HXmtkz6SsknUja6MoFUEJUCtja5rw/AV4ws/MUzVny\nYpb9W38eQc+01+lxC/iLmV3c9gCSJhMNEvgp4Bqiyahamdltkp4mqjt5RdEUngJuNbP/y3FtGbeR\ndG2OfVqUALVmNjHL+i4/zPgHlddhuIII3zx/R1SB3GIp8KHw+hx27Zv3BZJKQr3GPkS3ZJ4B/k3R\nsO9I2l9SVZ7jTANOkDRI0SRhFwMvtTOWZ4HWD1BJLR+Q/dg+xPTn07avZ8fJopYSzaOBpCOIbqNl\n8k/gOEn7hm2rwjX2BvqZ2VSiOqPD2u4oaZyZzTWz24lGaT6Q6Of1hbR6kOGS9mqza7Ztnif6HQwM\nywe0vTaL5mdZIumCsI0UTSwFUYnnovD6M1mu13VSnjBcId1BdP+9xc+IPqRnE02Tuyvf/pcTfdj/\nCbgqfLv/OVGl9huS5gH/R57Sc7j9dRPwAjAbmGFm7R22+zpgUqgAfhO4Kiz/b+BWSTPbxPEC0S2s\nWZIuJJrTZICk+USlg7ezxFpDlHgekjSH6HbUgUQf0E+FZX8Hbsiw+5dCZfocotFP/2RmzxLVv7wq\naS7R/B3piYxs25jZfOD7wEvh99gyxP7DwI2hYn0cUTK4PGwzn2hiJojqpK4Ox+xKs9w5Ynbcc845\n57yE4ZxzLhZPGM4552LxhOGccy4WTxjOOedi8YThnHMuFk8YzjnnYvGE4ZxzLhZPGM4552L5f2D1\ntkBuHw9xAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x9c84471a58>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "print(__doc__)\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.feature_selection import RFECV\n",
    "from sklearn.model_selection import StratifiedKFold\n",
    "\n",
    "# Create the RFE object and compute a cross-validated score.\n",
    "# The \"accuracy\" scoring is proportional to the number of correct\n",
    "# classifications\n",
    "rfecv_DoS = RFECV(estimator=clf_DoS, step=1, cv=10, scoring='accuracy')\n",
    "rfecv_DoS.fit(X_DoS_test, Y_DoS_test)\n",
    "# Plot number of features VS. cross-validation scores\n",
    "plt.figure()\n",
    "plt.xlabel(\"Number of features selected\")\n",
    "plt.ylabel(\"Cross validation score (nb of correct classifications)\")\n",
    "plt.title('RFECV DoS')\n",
    "plt.plot(range(1, len(rfecv_DoS.grid_scores_) + 1), rfecv_DoS.grid_scores_)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZIAAAEbCAYAAADwPQLqAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzs3Xd8lfX5//HXOyEh7A0yZAgoIiIq4t6jrop7jzpr62qt\n/Wprh/bnqqu1da9K1Tpw1EVFRQEVRPYIyN47rCyyr98f951wiEnODeSQdT0fj/PIucfnPtd9kpzP\n+WyZGc4559zOSqrpAJxzztVtnpE455zbJZ6ROOec2yWekTjnnNslnpE455zbJZ6ROOec2yWekTjn\nnNsljaKcJKkjcCTQBdgKzAImmVlJAmNzzjlXB6iqAYmSjgfuAtoCU4F1QBqwN9AbeAd4zMwyEx+q\nc8652iheRvII8E8zW1bBsUbAmUCymb2buBCdc87VZlVmJM4551w8kRrbJd0mqaUCL0maIumURAfn\nnNt1ku6R9FpNx+Hqr6i9tq4J20FOAdoAVwAPJSwq5xJA0hJJWyVlS1oj6RVJzWOOvyKpIDxe+rio\ngrSljydj0nYOv2StlpQl6QdJ90pqFj6/poJ4bpM0KUKsa8vH6lxtEjUjUfjzdOBVM0uP2edcXfJT\nM2sODAIOBH5X7vjDZtY85vFW+bQxj5sBJLUFxgNNgMPNrAVwMtCKoFPKMODKCmK5IjwWL9aDgMHA\nH8qfENYSeDd+V6Oi/gFOlvQZQUYyUlILwLv+ujrLzNYAIwkylF11O5AFXG5mS8LrLzezX5nZDOBV\n4ChJPUoTSOoPDATeiBDrSuB/wIAw7WhJ90v6FsgF9pLURdKHkjZKWiDp+nKXSZP0VlhamiLpgJhY\nukh6V9J6SYsl3borb4ZreKJmJNcSdAM+xMxygVTg6oRF5VyCSeoGnAYsqIbLnQS8V9m4KjNbAXxF\nUAIpdQUwwswyIsS6J8GXuKnl0t8AtACWAm8CKwjGep0PPCDphJjzhwLDCbry/wf4r6SUsDTzETAd\n6AqcCPxK0k/ixeVcqUgZSfgPshboL+kYYD+gdSIDcy5B/ispC1hOMC7qz+WO3yFpc/go/yH/35hj\nm2O+9bcDVsd53WGEGUn44X0ZVVdrlb0e8A0wBngg5tgrZpZuZkXAHgQDhu80szwzmwa8yPbVaZPN\n7B0zKwQeJxgPdhhwCNDBzP5iZgVmtgh4Abg4TmzOlYk6sv2vwEXAbKA43G3A2ATF5VyinG1mX0g6\nluCbeXtgc8zxR83sR20RsWkr2L8B6Bzndd8DnpZ0GNA0fHwSJdZKji2Ped4F2GhmWTH7lhK0q/zo\nfDMrkVRaejGgS5hhlUoGvo4Tm3NlImUkwNnAPmaWn8hgnNtdzGyMpFeARwn+vnfFF8A5ku6tonor\nV9I7BKWEJsCbZlawC68ZOwBsFdBWUouYzKQ7sDLmnD1Ln4Qlom5huiJgsZn13YVYXAMXtY1kEZCS\nyECcqwF/B06ObXjeSY8DLYFhpQ3qkrpKelzSwJjzhhGU7M8jfrVWZGa2HBgHPCgpLXzNa4HYsSMH\nSzo3nJHiV0A+8B3wPZAl6U5JTSQlSxog6ZDqis/Vf1EzklxgmqTnJP2j9JHIwJxLNDNbD/wb+FPE\nJB+VG0fyfnidjcARQCEwIWyDGQVsYfvG/LHhvhVmNrG67iN0CdCToJTxPvDnctViHxBkYpsI2mrO\nNbNCMysmmOpoELAYyCBoX2lVzfG5eizSFCmSrqpov5lV27cq55xzdVPkubYkpRLM+gswN+z94Zxz\nroGLWiI5jqBOdwnBiPY9gavMzHttOedcAxc1I5kMXGpmc8PtvYE3zOzgBMfnnHOulova2J5SmokA\nmNk8vBeXc845oo8jmSTpRbZ1J7wMqHDW0tqoffv21rNnz5oOwznn6pTJkydnmFmHeOdFzUh+AdwE\nlE7m9jXw9E7Gttv17NmTSZPqTL7nnHO1gqSlUc6LlJGEI9ofDx/OOedcmSozEklvm9mFkmay/ZQM\nAJjZwAqSOeeca0DilUhuC3+emehAnHPO1U1V9toys9KpsX9pZktjH8AvEx+ec8652i5q99+TK9h3\nWnUG4pxzrm6K10byC4KSx16SZsQcagF8m8jAnHPO1Q3x2kj+Q7BW9IMES+2WygpnPHXOOdfAxWsj\n2WJmS8zskrBdZCtB763mkrrvlgidi2jO6kyGT1pO1IlIXd2QX1TMu5NXkFdYHP/keuLr+euZsWJz\n/BNriahL7f6UYAxJF4J1rnsAcwjWbnf10DOjF7JlayF3ndYvYa9RUFRCVl4h7Zo3jpxmQ3Y+LZuk\nkJK87TvQkowcHv98Hh/NWIUZtG/RmOP36ZiIkHdYXmExD46YQ3JSEn/6af/d8pobsvNp1SSFRslR\nm0B3j7WZeXRs0RhJO5TuLx/N5vUJy9iQk88Nx/ROUHSJU1xibMjOp2PLtAqP3/3+TPbu1IKrjugJ\nQFZeITe+OpmWTVL46o7jSEtJ3o3R7pyof2n3AYcB88ysF3Aiwepqrh5am5nH3z6fx0vfLCIzL9pq\nAeuz8nnx60VkZG9bjbmkxHhr4jLWZeVVmOaBEXM47pHRrNq8tcLjb09aztINOWXbuQVFnPDYGO7/\nZE7Zvsy8Qs5++ls+n72WXxzbmy6t0nj6qwUVXa5a5BYU8bfP5/Hg/+bw4P/m8N+pKykuqbgEtHxj\nLuc/O45h45fy8reL+W7RhoTFVWrBumyO/OuXXPjceNZsqfh9r0xuQRGvjl/CrJVbqj2uUXPWcugD\no7j4+e+YvDR6rfj7U1fw+oRlpDZK4u1JK+pUabOkxBgxczWn/G0Mhz/0JVOXbfrRObNXZfL6hGU8\nMnIuW3KD/7V3Jq8gp6CY1Vvy+Ne3S6p8jXELMhi3ICMR4e+QqFOkFJrZBklJkpLM7CtJf09oZA6A\ndZl5tEhLoUlq9X4rmbx0I/d9Moc92zTlyD7tOGnfTmUlgxe/XkRBcbD0+Fc/rGPooK6Vx5eVx7Bx\nS3j5myVsLSxm/MINvHjVYCTx+oSl/PGDdI7q055Xrx2y3TfRLVsLeWvicrYWFvPH/84qS1Pq01mr\n+b93ZnDG/p156rKDAPhizjq2bC3kP98v45fH9aZjyzReHb+UzbmFfHDTkRywZ2s6tmjMPR/NZsKi\nDRy6V7vI70dxibE4I5se7ZptV9op762Jy3li1HwaN0rCDAqKS3jyqwVcf3QvVm7ayriFG1gdfoBv\nzCmgUbJ46tKDuO+T2Tw4Yg7/venISr+R5xcVk5tfTJtmqRUeX7l5Kx2aNya1UcXxlZQYv3tvBinJ\nSfywJosz//k1T156EIdV8D6YGQvXZ7NlaxEAM1ds5qnRC1mflU+Lxo34z/WHsX+3nVsksbjEWLV5\nK3u2bVq27/UJy2jTNIWF63M475nxXHBwNx46byDJSZWXTuatzeL3781iSK+2DB3Uhbvfn8XU5Zs5\nqHub7c57+ZvFvPTNYgBSksWNx/bm4iHbat5Xbt5KxxaNq/y9VrfMvEKufWUiE5dsok/H5rRrlsrv\n3pvJR7cctV0cw8YtITU5iez8IoaNX8LNx/fh3+OXcmD31rRtmsrTXy3gokP2pG0FfxPLNuRyzbCJ\nFBYbT116EKcO2GO33V95Ud/ZzZKaEywV+rqkJ4CcOGncLli1eSt3vTuDwx/6kpv/M6Varz1zxRZ+\n9vJEVm3eyvhFG7jz3Zmc9sTXLMnIYVNOAa9PWMZZB3ShQ4vGjExf86P0mXmFPDBiDqf8bQxD7h/F\n06MXcnL/Tvz82L0Y9cM6Pp6xmtVbtvLXT+fSvnljvlmQwbtTVm53jXcmr2BrYTHnHdSNUT+s45OZ\nq8uObdlayB8/SEeCz2avYWNOAQAfTltFm6YpFBWX8MLXi9haUMzL3yzm2L07cMCerQG4eEh32jdP\n5anRCyO9F2bGp7NWc+rfx3LS42MZdO9nXP2v73lvyooflTTMjLcmLmdgt1bMve805t53Ks9eHmRy\nd747kye/WkBhiXHYXu04vHc7zju4Kx/dfBRnDOzM7SfvzfQVW/h4xmqKS4wPp6/i/akryq5dUFTC\nRc99x/GPjWbFptwfxTl12SaOefgrTnx8dIWxAbwxcRkTl2zij2f254ObjqRlkxQue3ECL369qOyb\n/LgFGdzyxlQG3/cFJz0+lvOeGcd5z4zjno9ms1f7Zjx7+cG0bJLCFS9PYO6arLJrr96ylXcnr+DF\nrxdV2lYRvJdrOO2JsRz98Fd8PX89AGu25DF67jouO7QHY//vOG48tjfDJ6/grndnUFJJaa6kxPjN\n29Np1rgRT15yIEMHdaVJSjLDJy3f7rzlG3N58H9zaN88lcN7t6NNs1Tuem8md74zg0lLNnLFSxM4\n8qEvOenxMXwwbWWlrxdVSYnx6vglzFmdud19vzp+CV/+sBYzIye/iKv/NZFpyzfz0Ln7M/JXx3D/\nOfvzw5osnh+7qCzdppwC/jttJecd3JUT+3Xk5W8XMzJ9DYszcvjZET2587R+5BQU8eSXPy5hmxm/\nf38mjZKSGNClJbe8MYXRc9dtd87ijBzuGD6dTeH/TyJFLZEMJWho/zXBzL+tgL8kKqiG7r0pK7jr\n3ZkAHNS9NaN+WMe4BRkc0af9Ll97zupMrnh5Ai2bpDD8xsPp3CqNKcs2c92wiVz24gSO7tue3IJi\nbj6hDy2bNOLdySvJKywuq6eduyaLn786ieWbtnJE73acc2A3Tu7fkT4dW1BcYoxfuIF7P0pn384t\nKSop4Z0bj+I3w6dz3yezOW6fDrRv3rjsn/HgHm3463n7M29tFvd8mM7gHm3Zo1UaD/3vBzZk5/Pw\neQP57Tsz+GDaSs49sBtj5q3jysN7siE7n9cnLKNpaiM25BRw0/F9yu4vLSWZa47qxcOfzmXGis0M\n7Na67Njyjbm0bJJCqybBCgi5BUVc/+9JfLtgA707NOOen/Zn4focxs5fz+1vT+fp0Qu545S9OXVA\nZwDSV2Xyw5os/t/ZAwCQxKkDOnNy/z2YuXILvdo3K7t2eece1I2XvlnMgyPm8OSXC5i7NviQ3phT\nyLVH9eKBEXOYtnwzaSlJ3PT6FN6+8XAaNwre84KiEu56dybtm6fSMi2F29+ezuOfz+OYvTtw+F7t\naNc8lfyiEh4a8QNH9G7HBQd3QxIf3HQkvx0+g/s+mcOkJZvIyi/k2wUbaNcslaP7tueI3u3p1Cqo\nt2/TNIX9u7ZCEvt2bsGFz43np09+Q8u0RhSXGJtyt1VxfrdoA89cfjApyUkUFpcwZu56vlmQwdj5\n61m0Poe9OjSjc6s0HhjxA5/c0p53p6ygxOD8g7vRNLURd53Wj9Rk8Y8vF1Bi0LZZCuMXbWCfTi15\n9IKBSOKjGauYuXILf7vogLK2hTMGduaj6av545n9aZoafHQ9MnIuyUniuSsGs0erNIpLjL99Po8n\nv1rAW5OW06ZpCjcd35tRc9Zx25vTePHrxTx16UF0bxeUljLzCpm9KhMzSBIM7Na60tK/mXHvR+kM\nG7+UlmmNePOGw9m3cwseGDGHF74OSkQHdW9No6Qkpi7bxJOXHsTp+wd/Oyf378Tp++/BE6Pmc+qA\nPejdoTlvTVpOflEJVx3Rk9yCYs59ehx3DJ9OhxaNOW1AZ1IbJXHh4D15ZdxiPpwefBEb0LUVt5+8\nN/PXZvPNggz+39D9OGtQVy594TtueHUyJ+zTkSP6tGPO6kzenrSC1OQkzhjYOeFthlEXtuoFrDaz\nvHC7CdDJzJYkNLpqMnjwYKuts/+u3LyV4ZOWc+HgPenSugkfz1jFrW9M5bC92vHIBQfQrlkqJz42\nhrbNUvngpiNJqqIqoCrrsvJ4+quFvD5hKW2bpTL850eU/TMBzFq5hUte+I6svCJO6d+J568czDfz\nM7j8pQk8f8XBnLLfHnw6aw2/fmsazdMa8fRlB3FIz7Y/ep3ZqzL56ZPfUFxi/P70ftxwTG/mr83i\n9H98zbF7d+CxCwYxZdkmrn5lIv+45EDOOqAL6au2MPTJbykqMfbp1IK5a7O4/uhe3H1Gf8568hsK\nikq4+sie3PnuTD646UiapCZzyt+CxTmH9GzL2zcevl0MWXmFHP3wVxQXGzccsxcn9e/EC2MX8f60\nlezRMo1nLj+Yfnu04Lphkxi3MIN7z9qPS4Z0L2ucLv1m/djn81iwLpv7zxnAZYf24E8fzOKticv5\n/u6TKs0wqjJm3nquevl7erVvxq9P3pv/zVzN/2at4dwDu/Le1JVcc2QvhvRqy42vTebKw3vwl6FB\nhvXkl/N59LN5vHDlYE7s15FP09fw7uQVTFi8kez8orLrp6Uk8eltx9CzfbOyfWbGs2MW8cjIH2jT\nNJVfHt+Hyw7tHrcBd3FGDsPGLaEwrOLs1b4ZR/Ruz6SlG/nTB+mcMbAzJ+3bkb99Pp9lG3NJS0ni\nkJ5t+ekBXTj3wK58MnM1t705jccuOIB/fjmfji3TePvn235PZlb2AZyanETvjs2ZszqT353Wj58d\n2ZMTHxtDy7QUPr7lqLK/+e8Xb+TC58bz2AUHcN7B3ZixYjNnPfktNx/fhzt+ss928X81dx3z1mRx\n6aHdaZGWQkmJ8dGMVfzpg3TMjIfPH8iijByeG7OILVu3ZZIdWjTmlhP6cPEh3berQjQz/vrpXJ4d\ns5CLD9mTMfPWU1BUwmn778Fr3y3jisN60L9LS574Yj5rs/J4/MIDOOfAbtvFtC4rj5MeG0OJwXVH\n92L4pBV0a9OEt8L35ZLnv2P8og3cdmJffn1ysKr55twCnhmzkOy8IopLjJHpa9iUW0hqoyT279qK\n4T8/nKQksTGngEdGzmXsvPWs3LyV1OQkLj20O788vjcdW1TcyB+FpMlmNjjueREzkknAEWZWEG6n\nAt+a2SFx0p0KPAEkAy+a2UPljrcBXgZ6A3nANWY2Kzx2G3A9wdK+L5jZ38P994T714eX+b2Zjagq\njtqckdzzYTqvhPWkwTeuVRzYvTXDrhlS9q3rvSkruP3t6Txx8aAq2yvKW5KRw+ez1/LtwgzGL9xA\nUYlx4eBu3Hbi3uzR6sd/XJOXbgrq8s/dn357tKSwuCSoAtm3Ez/ZrxO/eH0KA7u14rnLD660BwoE\nbSyTlmziyUsPLPtwfmHsIu4fMYcWaY1o37wxOflFfHvXCWX1xfPWZvH57LWMW5hBYZHxyjWH0DS1\nEa9+t5Q//ncW3do0oVGS+OqO45DEDf+exGez1/LK1YdwXAXfthasy+LhT+fy2ey1ADRulMQlQ7rz\nxZy1rMvMp1/nFsxYsaXsQ6kixSXG1a9M5LuFG3jjhkO5+l8TOb5fR564+MDIv4PyFq3PpnvbpjRK\nTqKgqIQbXp3E6LnrOah7a9684XBSGyVx/yezeeHrxRy+VzsG92zDc2MWcfJ+nXjq0oO2u1ZRcQlz\nVmeRUxBkJt3bNqVL6yYVvu6SjBw6tGhMs8ZRKyEq9/zYhTww4gcA+nduya9O6sux+3QoK0FBUAU0\n9KlvWZyRQ3Z+EY9ecADnl3ufzYwlG3LZo2VaUBL7zxRGpq/ljP078+H0Vbx27aEc1bf9ducf/+ho\nsvOLOGW/PUhfuYXlm7Yy5rfH0SItWsa+bEMuN742mdlh1dQJ/TpyxWE9SEtJJjOvkBe/XsTEJZvY\nq0Mz/nPdYWX/J/8YNZ/HP5/HZYd2576zB7AoI4eLnhtPRnYB5x/cjYfPG0hSksgrDBrJe8Vk5rEW\nrMvm0ZFz+TSsMn7msoM4LSy1TFm2iT9/kM5LPxtc6Yd/Vl4hL32zmJHpa/nnJYPo07HFj97T5Ru3\nkpaatEsZSKnqzkimmdmgcvumm9kBVaRJBuYRTK+yApgIXGJms2POeQTINrN7JfUDnjKzEyUNAN4E\nhgAFwKfAjWa2IMxIss3s0biBh2prRmJmHPvIaDq2aMxeHZrxzuQV7N+1Fa9dd+h2/xglJcaZ//yG\nzLxCPrz5qAob3mKt2JTLE1/ML6tS2KtDM47u056rj+y13bfVKG5/exqfzlpDUbHRv0tLXrvuUJrv\n5IfRnNWZPPbZPL6Ys5bf/mSf7aqkKrNlayFD7v+C/KISbj2hD7efEnzzXL4xly/mrOVnR/Sssjvp\ntOWbmbBoA2cf2JVOLdPYlFPAbW9NY+y89dx39gAuP6xHla+/MaeAM//xNRtzC8grLOH16w7lyGqo\nYiyVV1jMa98t5axBXcr+8QuLS/jHqPl8MWcdc1Zn0rppCp/9+phq+WCoLu9NWUFaSjKn7rdHpaXk\ncQszuPSFCTRLTWbiH04q+2JUmay8Qs56Msh8jt27A8OuGfKjcyYv3cjTXy0sK439v6H7ccXhPXco\n9q0FxQwbv4TBPdowuFyp2sz48od13PrGVPZoFZSi3p+6kvs+mcO5B3Xl0fMPKLvfBeuy+Xr+eq48\nvGeVnQYqMmPFZr5fvJGrj+y1w2l3p6gZCWYW9wF8DpwVsz0UGBUnzeHAyJjt3wG/K3fOJ8DRMdsL\ngU7ABcBLMfv/CPxf+Pwe4I4ocZc+Dj74YKuN5q/Nsh53fmz/HrfYzMxWb95qWwuKKjz3u4UZ1vfu\nEXbEg6Ns+vJNlV7zo+krre/vR1jfu0fYvR+m24pNubsU48hZq63HnR/baX8fa5tzCnbpWqWWb8yx\n4uKSyOff+sYU63HnxzZ/bWa1vH5xcckOvS+Tl260Pr//xI58aNQOxV0dMrLybG3m1t36mtXp7vdn\n2JNfzo98/pzVW+yS58fbvDVV/64Liopt/tosKylJzO/ju4UZts8fRtjhD3xhPe782H752mQrLCpO\nyGvVZsAki5JHRDopqHr6DlgGLAfGAX3ipDmfoDqrdPsK4Mly5zwA/C18PgQoAg4G9iUozbQDmgLj\ngX/atoxkKTCDoFqsTbz4a2tG8vyYhdbjzo9t+cacSOdPX77JjnhwlPX9/Qh7Y8LSHx3/LH2N9f7d\nJ3b+M9/aqs27loGUKiousTe/X2obsvOr5Xo7Y8WmXBs+aXmNvb6Z2bgFGTZ1WeUZuKt/xs5bZ33v\nHmHX/Ot7yy9seJmIWfSMJFLVVqmwCzBmlh3h3POBU83sunD7CuBQM7s55pyWBG0oBwIzgX7A9WY2\nTdK1BBNG5gDpQL6Z/UpSJyCDYKqW/wd0NrNrKnj9G4AbALp3737w0qWRVozcrS5+fjybcgoZ+etj\nIqfZmFPAbW9O5ev5GVx8yJ7cc1YwucDns9fym7ens2+Xlrx27ZDIdcbOucptzCmgdZOUne7kUtdF\nrdqKN/vv5Wb2mqTby+0HwMyqWnp3JbBnzHa3cF8ZM8sErg6vKWAxsCg89hLwUnjsAYJ2FsxsbUwc\nLwAfV/TiZvY88DwEbSRV3WdNyMwrZNKSTVx/zF47lK5ts1ReuXoIj38+l6e+WsjYeevZkFNAflEJ\n+3ZuybCrD/FMxLlqEq890gXitZqW9g9tUeVZFZsI9A27Dq8ELgYujT1BUmsg14LeYNcBY8PMBUkd\nzWxdODnkuQRTtCCps21bcOscYNZOxFbjvp6XQVGJcUK/He/fnZwkfvuTfhzQrTXDxi/h1AGdObJP\nO47s075OzMvjnKtf4mUkpTOkzTaz4TtyYTMrknQzMJKg++/LZpYu6cbw+LMEbSHDJBlB9dW1MZd4\nV1I7oBC4ycxKp8J8WNIggqqtJcDPdySu2mLUD2tp3TSFA/dsHf/kSpyy3x6csl/NTYvgnHMQPyM5\nXdJdBD2udigjAbBgfMeIcvuejXk+Hti7krRHV7L/ih2No7ZZtD6bz2ev5YR+HWvdDK3OObej4mUk\nnwKbCNYfyYzZL8DMrGXCIqunlm/M5bIXJ5CanMRtJ/at6XCcc26XxVvY6rdm1hr4xMxaxjxaeCay\n49Zl5nHZixPILSjmtesOZa8OzWs6JOec22WRhiib2dBEB9IQPD16IWsy8xj+88PZt7Pnw865+qHK\nEomkb8KfWZIyw5+lj8yq0rrtFRWX8PGM1ZzYr2PZlOfOOVcfVFkiMbOjwp870/3XxRi/aAMZ2fmc\ndUCXmg7FOeeqVaQuQ5J6S2ocPj9O0q3hGBAX0YfTVtG8cSOO34lxI845V5tF7Xv6LlAsqQ/BaPE9\ngf8kLKp6Jr+omE/T13DKfp18wKBzrt6JmpGUmFkRwUjyf5rZb4HOiQurfhk9dz1ZeUVereWcq5ei\nZiSFki4BrmLb3FY+oVNEH05fRdtmqdW6joVzztUWUTOSqwnWF7nfzBaH82e9mriw6o/C4hK+nLOO\nUwfsUbYaoHPO1SdRx5HMBm6FsuVxW5jZXxMZWH2xcH02WwuLGVLB+ubOOVcfRO21NVpSS0ltgSnA\nC5KqmkLehdJXBsNt9uviAxCdc/VT1LqWVuH07ucC/zazQ4GTEhdW/ZG+KpO0lCSfDsU5V29FzUga\nSeoMXEglC0m5iqWv2kK/PVqS3EBXWHPO1X9RM5K/EKwrssDMJkraC5ifuLDqBzNj9upMr9ZyztVr\nURvbhxOzHomZLQLOS1RQ9cXyjVvJyitivy6tajoU55xLmEgZiaQ0gtUL9wPSSveb2TUJiqteSF+1\nBfCGdudc/Ra1autVYA/gJ8AYoBuQlaig6ov0VZkkJ4l99vA5L51z9VfUjKSPmf0RyDGzYcAZwKGJ\nC6t+SF+1hT4dmvv8Ws65ei3yFCnhz82SBgCtAJ/GNo70Vd7Q7pyr/yK1kQDPhyPa/wh8CDQH/pSw\nqOqB9Vn5rMvKp79nJM65ei5qr60Xw6djgL0SF079sa2h3XtsOefqtyozEkm3V3XczHyalAqUlBjv\nT12JhJdInHP1XrwSiXc32kFmxl8+ns0H01bxq5P60qqJz7bvnKvf4q3Zfu/uCqS+ePSzubwybgnX\nH92L207sW9PhOOdcwkWd/XdY7BrtktpIejlxYdVNmXmFPD16Iecc2JXfn74vks+v5Zyr/6J2/x1o\nZptLN8xsE3BgYkKquxasy8YMTt+/s2cizrkGI2pGkhR2/wUgXJckatfhBmP+2mCw/96dfMp451zD\nETUzeAwYL6l04sYLgPsTE1LdNX9tNmkpSXRr07SmQ3HOud0m6jiSf0uaBJwQ7jo3XH7XxZi3Lpve\nHZr72iPD69yYAAAgAElEQVTOuQYlcvVUmHF45lGF+WuzOGyvdjUdhnPO7VZR20h2iqRTJc2VtEDS\nXRUcbyPpfUkzJH0fzuNVeuw2SbMkpUv6Vcz+tpI+lzQ//Nmm/HVrQlZeIau35NGno7ePOOcaloRl\nJJKSgaeA04D+wCWS+pc77ffANDMbCFwJPBGmHQBcDwwBDgDOlNQnTHMXMMrM+gKjwu0aN39dNgB7\nd/IxnM65hiXqOJK/RtlXzhCCpXkXmVkB8CYwtNw5/YEvAczsB6CnpE7AvsAEM8s1syKCOb7ODdMM\nBYaFz4cBZ0e5h0TzHlvOuYYqaonk5Ar2nRYnTVdgecz2inBfrOmEGYSkIUAPgkWzZgFHS2onqSlw\nOrBnmKaTma0On68BOkW8h4Savzabxo28x5ZzruGJN2njL4BfAr0lzYg51AIYVw2v/xDwhKRpwExg\nKlBsZnPCEs9nQA4wDSgun9jMTJJVEvsNwA0A3bt3r4ZQqzZvXTZ9OnqPLedcwxOv19Z/gP8BD7J9\nW0SWmW2Mk3Yl20oREJQ0VsaeYGaZwNUACoaCLwYWhcdeAl4Kjz1AUKIBWCups5mtltQZWFfRi5vZ\n88DzAIMHD64ws6lOC9Zmcaj32HLONUBVVm2Z2RYzW0LQCL7RzJaa2VKgSFK8pXYnAn0l9ZKUClxM\nsChWGUmtw2MA1wFjw8wFSR3Dn90Jqr/+E573IXBV+Pwq4IP4t5lYWXmFrPIeW865BirqOJJngINi\ntrMr2LcdMyuSdDMwEkgGXjazdEk3hsefJWhUHxZWT6UD18Zc4l1J7QiW+b0pZq6vh4C3JV0LLAUu\njHgPCeM9tpxzDVnUjERmVlY9ZGYlkuKmNbMRwIhy+56NeT4e2LuStEdXsn8DcGLEuHeLBWtLMxIv\nkTjnGp6ovbYWSbpVUkr4uI2wLcPBoowcUpO9x5ZzrmGKmpHcCBxB0Fi+AjiUsEeUg/VZ+bRvnuo9\ntpxzDVLUSRvXETSWuwpkZOfTvkXjmg7DOedqRNSR7XtLGiVpVrg9UNIfEhta3ZGRnU/75p6ROOca\npqhVWy8AvyPoQYWZzcBLKGU2ZBfQrllq/BOdc64eipqRNDWz78vtK6ruYOoiM2NDjldtOecarqgZ\nSYak3oABSDofWF11koZhy9ZCCovNq7accw1W1HEkNxFMN9JP0kqCqUwuS1hUdUhGdj4A7Zt71ZZz\nrmGKm5FISgIGm9lJkpoBSWaWlfjQ6ob1WQUAdPASiXOugYpbtWVmJcD/hc9zPBPZ3oacoETSzjMS\n51wDFbWN5AtJd0jaM1zqtq2ktgmNrI7IyPKqLedcwxa1jeSi8OdNMfsM2Kt6w6l7MrILSE4SbZp6\nRuKca5iitpFcbmbf7oZ46pyM7HzaNkslyadHcc41UFHbSJ7cDbHUST6q3TnX0EVtIxkl6bxwFUMX\nY312gbePOOcatKgZyc+B4UCBpExJWZIyExhXnbHBSyTOuQYu6uy/vvRfBcwsrNryEolzruGK2msL\nSWcBx4Sbo83s48SEVHfkFBSTV1jiJRLnXIMWdRr5h4DbgNnh4zZJDyYysLpg2xgSz0iccw1X1BLJ\n6cCgsAcXkoYBUwmmlm+wSufZaudVW865BixqYztA65jnrao7kLooIzuYZ8tLJM65hixqieRBYKqk\nrwARtJXclbCo6ojSEkkHX4vEOdeARe219Yak0cAh4a47zWxNwqKqI0ozkra+OqJzrgGL2th+DpBr\nZh+a2YdAnqSzExta7ZeRnU+bpimkJO9IDaFzztUvUT8B/2xmW0o3zGwz8OfEhFR3ZGQV+PTxzrkG\nL2pGUtF5kceg1FcbcnwwonPORc1IJkl6XFLv8PE4MDmRgdUFGdkF3mPLOdfgRc1IbgEKgLeAN4E8\ntl+bpEHKyPJ5tpxzLmqvrRy8u+928ouKycov8qot51yD592NdlJOfjEALdJSajgS55yrWZ6R7KSc\n/CIAmqYm13AkzjlXs6rMSCT9Nfx5wc5cXNKpkuZKWiDpR1VjktpIel/SDEnfSxoQc+zXktIlzZL0\nhqS0cP89klZKmhY+Tt+Z2HZVbkFQImnWuMF3XnPONXDxSiSnh6si7vDkjJKSgaeA04D+wCWS+pc7\n7ffANDMbCFwJPBGm7QrcCgw2swFAMnBxTLq/mdmg8DFiR2OrDjkFXiJxzjmIn5F8CmwCBsaujBhx\nhcQhwAIzW2RmBQS9vYaWO6c/8CWAmf0A9JTUKTzWCGgiqRHQFFgV/bYSLzffSyTOOQdxMhIz+62Z\ntQY+MbOWZtYi9meca3cFlsdsrwj3xZoOnAsgaQjQA+hmZiuBR4FlwGpgi5l9FpPulrA67GVJbeLd\nZCJ4icQ55wKRGtvNbKikTpLODB8dqun1HwJaS5pGMFZlKlAcZg5DgV5AF6CZpMvDNM8AewGDCDKZ\nxyq6sKQbJE2SNGn9+vXVFO42uWFG0izVSyTOuYYt6qSNFwDfAxcAFwLfSzo/TrKVwJ4x293CfWXM\nLNPMrjazQQRtJB2ARcBJwGIzW29mhcB7wBFhmrVmVhwusvUCQRXaj5jZ82Y22MwGd+hQXfneNqXd\nf5s29hKJc65hi/p1+g/AIWa2DiAskXwBvFNFmolAX0m9CDKQi4FLY0+Q1JpgVuEC4DpgrJllSloG\nHCapKbAVOBGYFKbpbGarw0ucA8yKeA/VykskzjkXiPopmFSaiYQ2EL99pUjSzcBIgl5XL5tZuqQb\nw+PPAvsCwyQZkA5cGx6bIOkdYApQRFDl9Xx46YclDQIMWAL8POI9VKvSEkmTFC+ROOcatqgZyaeS\nRgJvhNsXAXG73YZdc0eU2/dszPPxwN6VpP0zFUxVb2ZXRIw5oXILimiamkxSkmo6FOecq1FR59r6\nraRzgaPCXc+b2fuJC6v2yyko9h5bzjnHDqwpYmbvETR6OyA3v4im3j7inHM+19bO8hKJc84FPCPZ\nSbkFRT6q3Tnn2IGqLUmpQD+C3lJzwy67DVZOfjEt0jwjcc65qAMSzwAWAv8AngQWSDotkYHVdrkF\nRT6GxDnniF4ieQw43swWAEjqDXwC/C9RgdV2OfnFPqrdOeeI3kaSVZqJhBYBWQmIp87wEolzzgWq\n/CQMx44ATJI0AniboI3kAoIpUBqsnAIvkTjnHMSv2vppzPO1wLHh8/VAk4REVAcUFZdQUFTiJRLn\nnCNORmJmV++uQOqS3MJw5l8fR+Kcc9Ea28PZfq8HesamMbNrEhNW7earIzrn3DZRPwk/AL4mmDq+\nOHHh1A2+OqJzzm0TNSNpamZ3JjSSOqSsROJtJM45F7n778eSTk9oJHVIWYnEe20551zkjOQ2gsxk\nq6RMSVmSMhMZWG3mqyM659w2UdcjaZHoQOqSsvXavY3EOeeqLpFI6hnnuCR1q86A6oLcsqotL5E4\n51y8T8JHJCUR9NqaTDAQMQ3oAxwPnEiwHO6KRAZZ2+SUNbZ7icQ55+INSLxAUn/gMuAaoDOQC8wh\nWIv9fjPLS3iUtUxZicTbSJxzLn4biZnNBu7eDbHUGTkFxaQki9RGvi6Yc875J+FO8PXanXNuG89I\ndkJOQbG3jzjnXMgzkp2QW1DkPbaccy4UdaldSbpc0p/C7e6ShiQ2tNorJ99LJM45VypqieRp4HDg\nknA7C3gqIRHVAbkF3kbinHOlomYkh5rZTUAegJltAlITFlUtl5NfTDOfZ8s554DoGUmhpGSCZXZL\n1ycpSVhUtZyXSJxzbpuoGck/gPeBjpLuB74BHkhYVLVcToGXSJxzrlTUSRtflzSZYEoUAWeb2ZyE\nRlaL+TgS55zbJu6nYVillW5m/YAfEh9S7VZSYuQWeq8t55wrFbdqy8yKgbmSuu/oxSWdKmmupAWS\n7qrgeBtJ70uaIel7SQNijv1aUrqkWZLekJQW7m8r6XNJ88OfbXY0rl2RV1SMmc/865xzpaK2kbQB\n0iWNkvRh6aOqBGFJ5ingNKA/cEk4AWSs3wPTzGwgcCXwRJi2K3ArMNjMBgDJwMVhmruAUWbWFxgV\nbu82PvOvc85tL+rX6j/uxLWHAAvMbBGApDeBocDsmHP6Aw8BmNkPknpK6hQTWxNJhUBTYFW4fyhw\nXPh8GDAa2G3ryW8tCDKSJt5G4pxzQMQSiZmNIWgfaRE+5oT7qtIVWB6zvSLcF2s6cC5AOFK+B9DN\nzFYCjwLLgNXAFjP7LEzTycxWh8/XAJ2ogKQbJE2SNGn9+vUR7jKanLJldr1E4pxzEH2KlAuB74EL\ngAuBCZLOr4bXfwhoLWkacAswFSgO2z2GAr2ALkAzSZeXT2xmRji2pYJjz5vZYDMb3KFDh2oINeCr\nIzrn3PaifhreDRxiZuugbEDiF8A7VaRZCewZs90t3FfGzDKBq8NrClgMLAJ+Aiw2s/XhsfeAI4DX\ngLWSOpvZakmdgXUR76FaeBuJc85tL2pje1JpJhLaECHtRKCvpF6SUgkay7droJfUOjwGcB0wNsxc\nlgGHSWoaZjAnEqzKSHiNq8LnVxEsA7zb+OqIzjm3vaifhp9KGgm8EW5fBPyvqgRmViTpZmAkQa+r\nl80sXdKN4fFngX2BYZIMSAeuDY9NkPQOMAUoIqjyej689EPA25KuBZYSVLXtNmUlEh/Z7pxzQPSR\n7b+VdC5wVLjreTN7P0K6EQRru8fuezbm+Xhg70rS/hn4cwX7NxCUUGqEl0icc257kT4NJfUCRpjZ\ne+F2E0k9zWxJIoOrjXIKvETinHOxoraRDGf72X6Lw30NTm5+ERKkNfKMxDnnIHpG0sjMCko3wucN\ncj2S7PximqYkk5Skmg7FOedqhagZyXpJZ5VuSBoKZCQmpNptc24BbZo1yDzUOecqFLXF+EbgdUlP\nEkwjv5xgbqwGZ2NuAW09I3HOuTJRe20tJBjX0Tzczk5oVLXYppwC2jT1jMQ550pFnSLlNkktgRzg\n75KmSDolsaHVTl4icc657UVtI7kmHHF+CtAOuIJw1t6GZlNOoZdInHMuRtSMpLSL0unAv80sPWZf\ng5FfVEx2fhFtm6XUdCjOOVdrRM1IJkv6jCAjGSmpBduPK2kQNucWAnivLeecixG119a1wCBgkZnl\nSmpHOGtvQ7IxJxhK09artpxzrkzUXlslBBMolm5vIJgBuEHZFGYkXiJxzrltolZtOYIeW4D32nLO\nuRiekeyAshKJV20551yZyHOhS0omWB+9LI2ZLUtEULXVxpygsb11U++15ZxzpaJOI38Lwdoga9nW\nW8uAgQmKq1balFtAy7RGpCR7Qc4550pFLZHcBuwTNrI3WBtzfFS7c86VF/Wr9XJgSyIDqQs2+cy/\nzjn3I1FLJIuA0ZI+AfJLd5rZ4wmJqpbamFPAHi3TajoM55yrVaJmJMvCRyoNdEErCHpt7du5ZU2H\n4ZxztUrUAYn3AjT0aeR95l/nnPuxqNPID5A0FUgH0iVNlrRfYkOrXbYWFJNXWOJjSJxzrpyoje3P\nA7ebWQ8z6wH8BnghcWHVPttGtfsYEuecixU1I2lmZl+VbpjZaKBZQiKqpXxUu3POVSxyry1JfwRe\nDbcvJ+jJ1WCUzfzrbSTOObedyCskAh2A98JHh3Bfg7Ep12f+dc65ikTttbUJuDXBsdRqvhaJc85V\nrMqMRNLfzexXkj4imFtrO2Z2VsIiq2U25RSQJGjZxBvbnXMuVrwSSWmbyKOJDqS225hbQOumqSQn\nNbil6p1zrkpVZiRmNjl8OsjMnog9Juk2YEyiAqttNuUU0sanj3fOuR+J2th+VQX7fhYvkaRTJc2V\ntEDSXRUcbyPpfUkzJH0vaUC4fx9J02IemZJ+FR67R9LKmGOnR7yHXeIz/zrnXMXitZFcAlwK9JL0\nYcyhFsDGOGmTgaeAk4EVwERJH5rZ7JjTfg9MM7NzJPULzz/RzOYCg2KusxJ4Pybd38xst1a3bcot\noEe7prvzJZ1zrk6I10YyDlgNtAcei9mfBcyIk3YIsMDMFgFIehMYCsRmJP2BhwDM7AdJPSV1MrO1\nMeecCCw0s6XxbiaRNuYUcGD31jUZgnPO1Urx2kiWAkuBw3fi2l0J1jEptQI4tNw504Fzga8lDQF6\nAN0IVmIsdTHwRrl0t0i6EpgE/CbsnpwwZhasReJdf51z7keiTtp4mKSJkrIlFUgqlpRZDa//ENBa\n0jTgFmAqUBzzuqnAWcDwmDTPAHsRVH2tZvuSUmzMN0iaJGnS+vXrdynIrPwiCovNMxLnnKtA1ClS\nniQoGQwHBgNXAnvHSbMS2DNmu1u4r4yZZQJXA0gSsJjtp145DZgSW9UV+1zSC8DHFb24mT1PMNkk\ngwcP/tEYmB2xITsYjNiuuWckzjlXXtReW5jZAiDZzIrN7F/AqXGSTAT6SuoVliwuBmIb7JHUOjwG\ncB0wNsxcSl1CuWotSZ1jNs8BZkW9h52VkR0sCtm+eeNEv5RzztU5UUskueEH/jRJDxNUKVWZCZlZ\nkaSbgZFAMvCymaVLujE8/iywLzBMkhGsdXJtaXpJzQh6fP283KUfljSIYKT9kgqOV7uMLM9InHOu\nMlEzkisIMoObgV8TVFmdFy+RmY0ARpTb92zM8/FUUkVmZjlAuwr2XxEx5mqTEc6z1b6FV20551x5\nUSdtLO16uxW4N3Hh1E4ZWflIPmGjc85VJN6AxJlUMFljKTMbWO0R1UIZ2fm0aZpKo+TITUrOOddg\nxCuRnBn+vCn8Gbuw1S71hKpLMrLzae89tpxzrkJRBiQi6WQzOzDm0J2SpgA/mj+rPsrILvCGduec\nq0TUuhpJOjJm44gdSFvnBSUSz0icc64iUXttXQu8LKkVIGATDWip3Q1eInHOuUpF7bU1GTggzEgw\nsy0JjaoWySssJju/yEe1O+dcJeL12rrczF6TdHu5/QCY2eMJjK1WWB8ORuzgJRLnnKtQvBJJs/Bn\ni0QHUluVTY/igxGdc65C8XptPRf+bHCDEEtlhBM2ehuJc85VLF7V1j+qOm5mt1ZvOLXPBp+w0Tnn\nqhSvamvybomiFiut2vL12p1zrmLxqraG7a5AaquM7AJapDUiLSW5pkNxzrlaKVL3X0kdgDsJ1lhP\nK91vZickKK5aY312vvfYcs65KkQdnf46MAfoRTD77xKChavqvYwsH9XunHNViZqRtDOzl4BCMxtj\nZtcA9b40ArAhp8C7/jrnXBWiZiSF4c/Vks6QdCDQNkEx1SoZ2fm0a+YlEuecq0zUubbuC6dH+Q3w\nT6AlwUqJ9VphcQmbcwu9ass556oQNSOZEM6vtQU4PoHx1Cobsn2JXeeciydq1da3kj6TdK2kNgmN\nqBbJ8MGIzjkXV6SMxMz2Bv4A7AdMlvSxpMsTGlkt4BmJc87FF3lxKjP73sxuB4YAG4F6P1hx2zxb\nXrXlnHOViZSRSGop6SpJ/wPGAasJMpR6zUskzjkXX9TG9unAf4G/mNn4BMZTq2Rk5dMkJZlmjaO+\nTc451/BE/YTcy8wsoZHUQn06NuesA7rUdBjOOVerRV1qt8FlIgAXD+nOxUO613QYzjlXq0VubHfO\nOecq4hmJc865XRK119bDYc+tFEmjJK1vCONInHPOxRe1RHKKmWUCZxJMId8H+G2ignLOOVd3RM1I\nShvlzwCGh/NuOeecc5G7/34s6QdgK/CLcMXEvMSF5Zxzrq6IOtfWXcARwGAzKwRygKGJDMw551zd\nELWx/QKC1RGLJf0BeA3wkXrOOedQlLGGkmaY2UBJRwH3AY8AfzKzQxMdYHWQtB5YuoPJ2gMZCQhn\nd/P7qH3qy734fdQ+1X0vPcysQ7yToraRFIc/zwCeN7NPJN2306HtZlHeiPIkTTKzwYmIZ3fy+6h9\n6su9+H3UPjV1L1F7ba2U9BxwETBCUuMdSOucc64ei5oZXAiMBH5iZpuBtvg4Euecc0TvtZULLAR+\nIulmoKOZfZbQyGre8zUdQDXx+6h96su9+H3UPjVyL1Eb228DrgfeC3edQ9BW8s8Exuacc64OiNxr\nCzjczHLC7WbAeDMbmOD4nHPO1XJR20jEtp5bhM9V/eHUPEmnSporaYGku2o6nqgk7SnpK0mzJaWH\npUgktZX0uaT54c82NR1rFJKSJU2V9HG4XVfvo7WkdyT9IGmOpMPr4r1I+nX4dzVL0huS0urKfUh6\nWdI6SbNi9lUau6Tfhf//cyX9pGai/rFK7uOR8G9rhqT3JbWOObbb7iNqRvIvYIKkeyTdA3wHvJSw\nqGqIpGTgKeA0oD9wiaT+NRtVZEXAb8ysP3AYcFMY+13AKDPrC4wKt+uC24A5Mdt19T6eAD41s37A\nAQT3VKfuRVJX4FaCmS0GAMnAxdSd+3gFOLXcvgpjD/9nLgb2C9M8HX4u1Aav8OP7+BwYENYOzQN+\nB7v/PqI2tj8OXA1sDB9Xm9nfExVUDRoCLDCzRWZWALxJHZkKxsxWm9mU8HkWwQdWV4L4h4WnDQPO\nrpkIo5PUjWDM0osxu+vifbQCjiH80mVmBWGvxzp3LwRjzppIagQ0BVZRR+7DzMYSfG7Fqiz2ocCb\nZpZvZouBBQSfCzWuovsws8/MrCjc/A7oFj7frfcRd0BimIulh9+opiQqkFqiK7A8ZnsFUCdG78eS\n1BM4EJgAdDKz1eGhNUCnGgprR/wd+D+gRcy+ungfvYD1wL8kHQBMJihp1al7MbOVkh4FlhFM3PqZ\nmX0mqU7dRzmVxd6V4AO51IpwX11wDfBW+Hy33kfcEomZFQNzJfni5XWApObAu8CvwjVkyljQsyJ+\n74oaJOlMYJ2ZTa7snLpwH6FGwEHAM2Z2IMFkp9tV/9SFewnbD4YSZIxdgGblF7arC/dRmboceylJ\ndxNUb79eE68fdYqUNkC6pO8J/hkAMLOzEhJVzVkJ7Bmz3S3cVydISiHIRF43s9Ku2msldTaz1ZI6\nA+tqLsJIjgTOknQ6kAa0lPQade8+IPgWuMLMJoTb7xBkJHXtXk4CFpvZegBJ7xHMBl7X7iNWZbHX\nuc8AST8jWHTwRNvWDXe33kfUxvY/EgT6F+CxmEd9MxHoK6mXpFSCxqoPazimSCSJoC5+TtimVepD\n4Krw+VXAB7s7th1hZr8zs25m1pPg/f/SzC6njt0HgJmtAZZL2ifcdSIwm7p3L8uAwyQ1Df/OTiRo\ng6tr9xGrstg/BC6W1FhSL6Av8H0NxBeJpFMJqoHPCgeOl9q992FmlT4IltQ9soL9RwG9q0pbVx/A\n6QS9HxYCd9d0PDsQ91EExfMZwLTwcTrQjqBXynzgC6BtTce6A/d0HPBx+LxO3gcwCJgU/l7+S1C6\nr3P3AtwL/ADMAl4FGteV+wDeAFYDhQSlxGurih24O/z/nwucVtPxx7mPBQTtuqX/88/WxH1UOSAx\n7MP/OzObWW7//sADZvbTShM755xrEOJVbXUqn4kAhPt6JiQi55xzdUq8jKR1FceaVGcgzjnn6qZ4\nGckkSdeX3ynpOoI+8c455xq4eG0knYD3gQK2ZRyDgVTgHAt6pTjnnGvAos7+ezwwINxMN7MvExqV\nc865OiPqXFtfmdk/w4dnIm6HSTJJj8Vs3xFOAFod135F0vnVca04r3NBOIPvVxUceyScHfeRnbju\noHAAZq0lKXsn0529MxOf7uzruZrh66673SUfOFdS+5oOJFY4CWFU1wLXm9nxFRy7ARhoZjuzBPUg\ngjE/kSlQF/5/zyaYSdvVY3XhD9HVD0UEy4D+uvyB8iWK0m+jko6TNEbSB5IWSXpI0mWSvpc0U1Lv\nmMucJGmSpHnhfF2la5o8ImliuF7Dz2Ou+7WkDwlGmpeP55Lw+rMk/TXc9yeCQZ8vlS91hNdpDkyW\ndJGkDpLeDV93oqQjw/OGSBqvYJ2VcZL2CWdQ+AtwkaRpYfp7JN0Rc/1ZknqGj7mS/k0wMHBPSaeE\n15wiaXg41xrhezU7vO9HK7jHY8PXmxbG0yLc/9uY9+vein6RlZ0j6cpw33RJr0o6AjgLeCR8nd7h\n41NJk8PfQb8wba/wPmZKuq+i13W1WE2P1vRHw3gA2UBLYAnQCrgDuCc89gpwfuy54c/jgM1AZ4KR\n1CuBe8NjtwF/j0n/KcEXo74Eo37TCEoJfwjPaUwwwrxXeN0coFcFcXYhmBKkA8FcdF8CZ4fHRhOs\nyVHh/cU8/w9wVPi8O8G0NYT33yh8fhLwbvj8Z8CTMenvAe6I2Z5FMG6rJ1ACHBbubw+MBZqF23cC\nfyIYtT2XbW2grSuI9yPCWSsIMsFGwCkEmb3C9/Jj4Jhyv5MKzyFY92Ie0D48r20lv9tRQN/w+aEE\nU+BAMKXHleHzm2LfT3/U/seOFOud2yVmlhl+m76VYDryKCZaON23pIXAZ+H+mUBsFdPbZlYCzJe0\nCOhH8KE3MKa004ogoykAvrdgnYbyDgFG27YJCl8n+KD8b8R4Icgk+ktli4i2DEsKrYBhkvoSTGeT\nsgPXLLXUzEqnBz+MoNro2/C1UoHxwBYgj6D09DHBh3153wKPh/f3npmtkHQKwXs2NTynOcH7NTYm\nXWXnHAAMN7MMADMrv/5H6czURwDDY96bxuHPI4HzwuevAn+N+064WsMzkv/f3t2zRhFFYRz/PylE\niBCIYmNnRNIpKoJYKH4AQTBosBFtIhoVQfALSFQQLGyCgqVirYZEMAYURfAlibGwMdjaCDYKkmNx\n7iaTJRs1g9Hg86tmZ+7Mzuwuc+bek9xjy+0aWdfmVmXdd8owaxn3X1XZ9q2yPFN5PcP832/znx8G\n+dTcHxHD1Q2S9lKZxfoPaCN7DV+b3vc6MBoRB5Q1Yx632H/28yhWV5ar5y3gYUT0Nh9A0k5ycsWD\nwClgX3V7RFySdJ/MzTxVlmIVMBARg4tc24JtJPUvsk9DG/A5Ira22L6ip3L/nzlHYsuqPKneJRPX\nDdPA9rK8n6U9qfdIait5k43k0M4wcEI5vT6SNktq/8lxXgB7JK1TFnXrBcZ+81xGgNkbq6TGjbOD\nuah8CdMAAAFRSURBVKm8j1baf2F+Ea9pso4JkraRw3ELeQ7slrSptG0v17gG6IiIB2ROakvzjpK6\nImIyIi6Ts153k5/XsUqeZYOk9U27tmrziPwO1pb1nc3XFlkf54OkntJGyoJfkD2kw2X5SIvrtX+U\nA4n9DVfJ8f2GG+TNexzYxdJ6Cx/JIDAE9JXewE0ymf5K0ltgkJ/0wssw2gVgFBgHXkbE706PfhrY\nURLP74C+sv4KMCDpddN5jJJDYW8kHSJrynRKmiJ7E+9bnOsnMiDdljRBDmt1kzfue2XdE+DcAruf\nLUn8CXI22aGIGCHzO88kTZL1U6oBjlZtImIKuAiMle+xUcrgDnC+JPS7yCBxvLSZYq6U9RngZDnm\nSqlIaMUv/UOimZlZK+6RmJlZLQ4kZmZWiwOJmZnV4kBiZma1OJCYmVktDiRmZlaLA4mZmdXiQGJm\nZrX8ANyN7OlrhsU1AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x9c84c28dd8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "rfecv_Probe = RFECV(estimator=clf_Probe, step=1, cv=10, scoring='accuracy')\n",
    "rfecv_Probe.fit(X_Probe_test, Y_Probe_test)\n",
    "# Plot number of features VS. cross-validation scores\n",
    "plt.figure()\n",
    "plt.xlabel(\"Number of features selected\")\n",
    "plt.ylabel(\"Cross validation score (nb of correct classifications)\")\n",
    "plt.title('RFECV Probe')\n",
    "plt.plot(range(1, len(rfecv_Probe.grid_scores_) + 1), rfecv_Probe.grid_scores_)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYwAAAEbCAYAAADJWrOxAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XecXHXZ///Xe3ezqZteCOmEhBAgJBACoSNFioKANEGU\nIuINgnoDghXuLypI+YmCYkSaNKWKdOmIgfQKCQnphfTNtmR3yvX745zZTDazOyfJTmZ3cz0fj3ns\nnDrXmd09n/PpMjOcc865bAryHYBzzrnmwRMM55xzkXiC4ZxzLhJPMJxzzkXiCYZzzrlIPMFwzjkX\niScYzjnnIimKspOknsARwJ7AJmAWMMnMkjmMzTnnXBOihjruSToOuBHoCkwFVgNtgKHAYOAZ4C4z\nK8t9qM455/IpW4JxB/AHM1uSYVsR8BWg0MyezV2IzjnnmoIGEwznnHMuJVKlt6RrJXVU4K+Spkg6\nKdfBOeecazqitpK6NKynOAnoAnwTuC1nUTnXyCQtkrRJUoWkLyQ9LKlD2vaHJdWE21Ov8zIcm3rd\nm3Zs7/BBaqWkcklzJN0iqX34/tIM8VwradIOxnq9pFnhZy2UdH2d403S3o3xvTmXLmqCofDnqcDf\nzGx22jrnmouvmlkHYCQwCripzvbfmlmHtNff6x6b9roaQFJXYDzQFhhrZiXAiUAngoYhjwAXZ4jl\nm+G2HYlV4Tm7ACcDV0s6P8oX4NzOiNSsFpgs6Q1gEHCTpBLAm9S6ZsnMvpD0OsHNeGf9CCgHLko1\nMzezpcAPACStB/6fpAFmtjhcNxwYATy5I7Ga2W/Tdpkr6Z8Ezd6faoTrca5eUXMYlxE0rz3EzKqA\nYuCSnEXlXA5J6gucAsxvhNOdADxXX58kM1sGvEOQo0j5JvCKma3d2VglCTgKmL2dcTu33SIlGOE/\nwypguKSjgf2AzrkMzLkceEFSObCUoE/RL+tsv05SafiqezN/IW1bqaTvhOu7ASuzfO4jhAmGpALg\nQhoujooSa8rNBP/HD2U5n3M7LWpP79uB84BPgES42oD3cxSXc7nwNTN7U9IxwBNAd6A0bfudZvaz\nho7NsH4d0DvL5z4H/FHSYUC78PXyTsaKpKsJ6jKOMrPqLOdzbqdFrcP4GrCP/1G6lsDM3pP0MHAn\nwd/2zngTOFPSLQ0US1VJeobg5t4WeMrManYm1rDl1Y3A0WGxl3M5F7UOYwHQKpeBOLeL/Q44UdKB\nO3meu4GOwCOSBgBI6iPpbkkj0vZ7hCCXfjbZi6MajFXShcCvgRPNbEE9xxRLapP2KtzOz3RuG1ET\njCpgmqQ/S/p96pXLwJzLJTNbAzwK/CLiIf+q0w/j+fA864HDgRjwcVjv8Bawka0rqt8P1y0zs4k7\nGeutBHUnE9Piub/OYbMJBgpNvbyRittpkYYGkfStTOvNbHuflJxzzjVTkceSklRMMEotwFwzi+Us\nKuecc01O1BzGsQTlrosIepn2A75lZt5KyjnndhNRE4zJwDfMbG64PBR40swOznF8zjnnmoiold6t\nUokFgJl9hreacs653UrUfhiTJD0APBYuXwhkHGkzn7p3724DBw7MdxjOOddsTJ48ea2Z9Yiyb9QE\n43vAVcA14fIHwB93ILacGjhwIJMmNbl0zDnnmixJi6PuGynBCHt43x2+nHPO7YYaTDAk/cPMzpU0\nk2DsqK2Y2YgMhznnnGuBsuUwrg1/fiXXgTjnnGvaGmwlZWapYZv/x8wWp7+A/8l9eM4555qKqM1q\nT8yw7pRsB0k6WdJcSfMl3ZhhexdJz0uaIWmCpP3Ttv1Q0uxw7uInJbWJGKtzzrkcaDDBkPS9sP5i\nn/CmnnotBGZkObYQuI8gYRkOXBBOTZnuJ8C0sC7kYuCe8Ng+BC2yRpvZ/kAh4HMWO+dcHmWrw3gC\neBX4DcHY+ynl4SidDRkDzE8NvyzpKeAMgkmYUoYDtwGY2RxJAyX1SoutraQYwYQzKyJcj3POuRxp\nMMEws40EQzJfACCpJ9AG6CCpg5ktaeDwPgTTS6YsAw6ts8904CzgA0ljgAFAXzObLOlOYAnB0Mxv\nmNkb0S+redocS/D67C/4fHUFAB3aFPGNQwfQoXXU7jKuPitKN1FVEwegb5d2tGnl00M0NbFEkrJN\nMbp1aJ3vUFw9ok7R+lWCPhh7EswvPAD4lGBu751xG3CPpGnATGAqkJDUhSA3MohgWsqnJV1kZo/V\nPYGkK4ArAPr377+T4eRHVU2ce96axz8mLmVDVTAIsARmMGHhesZ9czQFBcpzlDtmcyzBDc/MoF1x\nIRceOoAD+nbCzFhfWUOXdsVbXdcH89awZH0VA7q2Z3DP9vTu1HanPz+RNH772hz+/P6WeYb6d23H\nC1cdQdf2xTt9/l1tY1WMz1aXs3hdFavKNteuH9CtHUcM7k6XHbimeCLJM5OX8fLMlfz26yMyfu/J\npFEdT9K2ePsT2ngiSdKguKjhKtObnpvJM5OXMaRnB44a0oN9e5cwoFt79tmjhE5td+1IRHO+KOP1\nWatYtK6SpeurGNmvMz86aSjtiqM/vNXEk7w9ZzU9O7bmoP5dtjuGeCJJLGE79J3nStSrvxU4DHjT\nzEZJOg64KMsxywlGtU3pG66rZWZlhBO7SBKwkGB2vy8DC8OJY5D0HMEkNdskGGY2DhgHMHr06Ghj\ntTchNfEkVz42hf/MW8OX99uDiw4bwOGDuyGJhz9cyM3/+oS7//0Z1315n10W0+ZYgtZFBQS/kuji\nieAfpH+3dgzboyPxRJKrn5jCW3NW07qogKcmLqVP57asr6xhUyzBAX06Me7ig+ndqS0PfLCAW1/+\ndKvzHTqoKxcdNoDj9+25zT9qImnMWFbKrOUb6VHSmv5d2zOkVwdaFW65KW3cFOPap6by7tw1XDCm\nH4cP7k755jg3/2s2Vz0+hUcvG7PV/plU1cR5ZeYXHD20Oz1LorW7SCaNmkSy0XIxZsaEhet5/OMl\nvDprJbFE5j9zCQ7u34Xbzj6AvXuW1Hu+iuo4z09dzuaaBPGk8eyUZcwPc7X3vj2fX515wFb7f7Fx\nM9/868csXFvJqP6dOWLv7gztVUL/ru0oKhSL11WxtqKaE4f32uo7SiaNF6Yt5643PqO0qobvHjOY\ny48alPGmO31pKc9MXsZx+/QgnjQe/3gx1fFgxtuObYp46oqxDN+zIwBL11cxf3UFxw3ruc15lq6v\n4rZX51BVE2dAt/YM7VXCaSN6N5jgrC7fzMszVhJPGPGk8danq5i0eAMS7NmpLT07tuaB/yzkzU9X\ncde5B3LwgK61x26sivGPSUtZsLaCJeurEKJ/t3a0KSrkxekrWFtRTed2rXjvuuPo1K7+GOKJ4FqL\nwr/H9ZU1fPuhCSxZX8WtX9ufr4zYEwhyYdOWlvLBZ2uYuGgDJW2KGNCtHQO7t+fCQwfUe/7GEnW0\n2klmNlrSdGCUmSUlTTezeqe3lFQEfAYcT5BQTCQY8XZ22j6dgSozq5H0HYLJ7C+WdCjwIHAIQZHU\nw8AkM/tDQ3GOHj3amtPQIMmk8YO/T+PF6Su47awDOH/M1jkkM+PGZ2fy90lLufOcAzn7oD7bfRPf\nXi9OX8ENz0znkIFduf3sEezZedunzVTC8MK05XRqW8zRQ7ojwZ1vfMb81RVIcOaoPiSSxj+nreD/\nfW1/Tj9wT16YupyPF65jj45t6dKuFX9+fwFtWhVy8v69eOyjJZyy/x785NR9WbZhE1OWbOCpiUtY\nun4TAD1KWtOnc1uKCoQB81aVU7Y5vlVc++3ZkSe+cxid2rZi46YY54/7iHmryrnljP22+md6dvIy\n/vfp6Xxr7ABuPn2/jN9pTTx46v7/3vyMNeXVdGnXilu/dgCnjejN4nWVTFmygZH9ujCoe/vaY9aU\nV/OPSUt54uMlrKmo5rQDenPhof0Z1b8LhWFOakXpJiYuWs/QXiXs27vjVp9pZjw9eRnvzFnNFUfv\nxaj+XVhVtpkbn53BO3PXUNKmiLMP6ssx+/RgQNd27Nm5LQUSSTNmryjjP/PW8rePFlEdS3LPBSP5\n0rBemAU5g1TiVVpVw7cemsj0paW1n7tX9/bccPIw3p+3hmcmLeO9G46tzWUsXV/FNx74iPUVNZx7\nSD8mLdrArBUbyXTbaFdcyOVH7cVRQ7rz3/nreGXmSuauKme/PTvSu1Nb3vx0FT1KWvPtwwdy7uh+\n9ChpXXvd59w/nkXrKnnnumMpadOKWCLJ8g2bWLi2kp88P5NYwnj2e2NZun4TVz0xhY2bYjxy6RiO\nGdqj9hxPTVzKrS99giT6d23H4nWVVNYkaNOqgK+O2JMeJa1ZvL6K6liCrx64JyfvvwdvzF7Fz/85\ni9KqLdP7DOzWjgsPHcDXD+5bm2Mb//k6rnt6Ois2buKsUX350UlDmbeqnB8/O4NVZdV0a19Mv67t\nMGDxukrKNsX40rCeHLtPT37xz1lccsQgfv6Vum1+gu/38Y+X8PSkpSTNuOb4IZy03x5c8tAEFq2r\nYnCPDny6soyThvciafDRgnVUVMcpEOy3Zyc2xxIsWV9F1/bFjL/p+G1/KRFImmxmoyPtGzHBeJNg\nAvrfAN0JiqUOMbPDsxx3KsF8xIXAg2b2K0lXApjZ/ZLGEsyzYQRTSl5mZhvCY28hmAM5TlBUdXk4\nREm9mkOCMX91efgElKBsc4xZy8u44eR9+J9j9864f3U8wTcfmMCERes5Yu9u3HTKvuzfp1Ojx5VI\nGne+MZc/vfs5w3t3ZOHaSooKxfeOHUyXdsUkzVhVVs2SdZV8vHA9KzdupnuH1myOJaioDm7ce3Vv\nzw9OHMrsFRt56MNF1MST/PCEoVx7wpCMnzlvVTmXPzqJxeuqOOugPvz27BG1T1gQJKj/mb+WGctK\nWbyuii/KNpMM/177dG7LkUN6cFD/zmyojDF9WSm3/Gs2o/p1YdzFB3PFo5OZunQDf7l4NMfus+2T\n6K0vfcID/1lI21aF9O/ajm4dipEgnjBWbNzEitLNJJLGwQO6cOkRgxj3/udMXxbkZtaUb/kzPHLv\n7hzQtxP//XwdM5eVkjQ4fHA3BnZvz7+mraC8Ok5xYQF9u7QFwYI1lbXHHjygC2cd1IehvUro3LYV\nt706h7fmrKa4sICaRJIvDevJ5MUb2BxLcP2X9+HCQwdkLZ5YXrqJKx6dxCcryxjcowPLNlRRE09y\n3D49Ofvgvvz+rXksWFPJ7y8YyZFDgptt++JCJLF0fRXH3fkuFx0WJKSfrizjkocmsimW4JFLxzCy\nX2cAyjfHWLK+iiXrqognjQHd2tGqsIB735nPyzOCrlsSjOjTiUuPHMRXR+xJQYGYtGg9d//7M/77\n+TpaFao2R72uooarnpjCb846gAvGbFusPH91OefcP56iwgLWV9YwuEd74kljU02C1394NO1aFfLj\nZ2fy7JRlHD64G3eccyB9OrfFwoT08Y8X88LUFcQSSfp2aUssYSwv3USH1kVUVMcZ2a8zt589gj5d\n2m71fdRVUR3n92/N4+H/LgKDmkSSIT07cPe5Izmg79b/k7FEsjb3etNzM3hm8jLe+OExWz1gPD1p\nKTc+NxMz44R9e1FZE+fD+esoELRpVcgDF49mzKCu3P/e5/z+rfn06tSao4f04Kgh3Rk7uHttrimZ\nNDZuiu1QcWTwu2r8BKM9wZN+AcFItZ2Ax81s3Q5FmCNNPcFYtqGKr/9pPJvjCYb07ADA8fv24rtH\n79VgzqE6nuDxj5bwh7fnUbopxh8uGFWbRW0M5Ztj/OCpabw1ZzUXjOnPLafvx8qNm7ju6elMXLSh\ndr8CQe9ObRm2RwnnjO7HCfv2xIBpS0tZV1HN8fv2qv0nWV66idnLN3Li8F4NXtvGqhjjF6zjpOG9\ndrqe5l/TV3DNU1MpaV1EeXWc358/iq8emPl7SiSN56YsY84XQX1AaVVNeI1ij05tGNCtHQcN6MKx\nQ3sgiXgiyV8+WMjM5aUctlc3RvXrwrtzV/PkhCWsKq9mZL/OHLl3d04fuSeDewS/28rqOK/P/oK5\nq8pZsi64cY8d3I1DBnZl4qKgiGnh2i0JSOuiAn588jDOGd2Xv/5nIePeX8DQXiXcde6BteeMYlNN\ngttfm8OyDZsY0K0dhQXiuSnLWVtRTZtWBfzl4tEcNSTz4KQ3PDOdF6at4BdfGc6vX/mUkjZFPHzJ\nmG1yQ/WZuWwjy0urOHRQt3pvYPNXV/DEx0t4ZvJSyjYHT8tDe5Xw8jVH1ebE6pq2tJRvPvAxYwd3\n4+7zRjJ/dQVn/fFDzhzVl02xoNjw2uOHcO3xQzL+HVXHExRKFBUW1D6IvDB1OfvsUcJlRw7a6kEl\nm+Wlm7j37Xl0bV/M9780JGvR4+ryzRx3x7scsXd3xl0c3JcfHb+IX/xzNkfu3Z07zgnqjcyM9+et\n5fGPFvPdYwZz8IAt9R7xRHK7YtweuUgwBgErzWxzuNwW6GVmi3Ym0MbW1BKMV2au5Lqnp/OlYT05\n66A+3PrSp6ypqOYf3x0b+R8wXdnmGN9+cAKfrarglWuOon+3dg3uXx1P8NhHS9hQWcM1xw+prXR8\necZKxi9YyyEDuzKgW3uue3o6i9ZW8suvDuebYwfWHm9mrCmvJhn+iXRtX5y14jLf/jZ+ETf/6xNu\nrnMtuZJIGjU7WBmcTBqL1lWyZH0VK0o3M3Zwt62eQDfVBHVJjdHgIVUB27dL2wZzqIvXVfKlu94j\nkTRG9e/Mny86mJ4dc9NndlNNgpdmrOCVmSv5/vFDslYMb44ltro53/7aHP707ucA/Oy0fbn8qL1y\nEmdjuO+d+dzx+lx6lrSmd6c2TF+2kRP27cW93xiV9xZ7uUgwJgGHm1lNuFwMfGhmh+xUpI2sKSUY\nS9ZVcdrvP6BL+2JKq2oo2xyndVEBj11+KIcM7Jr9BPVYuj4476Du7Xn6ysMz3sDLN8d4ffYqfvfm\nZyzbENQBHDKwC/d94yD+8sEC/vLBwtpiD4Au7Vpx34UHcfjg7jscV1NSVRPfrtYsbmv3vTOfdRU1\n3HDyPnm/mTWkOp7guqdncNSQ7pw7ul/2A/KoJp7ksY8W8+nKMhavr2J474789LR9sza62BVykWBM\nM7ORddY1WOmdD00lwaiJJznnz+NZsCbICXTv0JrXZq+kX5d2jN6JxCLl1Zkr+d7jU/jSsJ4cPrgb\ne3Zuy7qKahavq2L6slKmLCklkTT227MjN52yL+sqq7nhmRlYWO568dgB/OTUfZnzRTkzlpVy3D49\n6de14dyKc65l2p4EI+pj2BpJp5vZi+EHnAGs3dEAWzIz47ZX5zB9aSl/uvCg2hvxmaP6NtpnnHJA\nb6750t489OEi3p6zunZ966IChu1RwpXH7MVRQ3owZmDX2uKMvbp34JcvzuKc0f1qKxZH9utcW5Hp\nnHPZRM1hDAYeJ+i4J4Ie3Beb2fzchrd98p3DMDN+8+ocxr2/gG8fPpCbT9/Zfo3ZP29DVYwVpZvo\nUdKaniWtc97s1jnXsjR6DsPMPgcOk9QhXK7YifhapGTS+Pk/Z/H4x0u4eOwAfpGhzXVjk0TX9sXN\nsseyc675yTbj3kVm9pikH9VZD4CZ+ZStoVdmreTxj5fw3WP24saTh/mTvnOuxcmWw0jVhNY/zoAD\ngp6gJa2LuOHLnlg451qmbAnG4PDnJ2b2dK6Dac4mL97AyP6d6+145JxzzV22RsCnhoMC3rQrgmmu\nNm6KMXdVOaMH7HyTWeeca6qy5TBeAzYQzH9RlrZegJnZ9ndXboGmLtmAGYweuP1DGDvnXHPRYA7D\nzK43s87Ay2bWMe1V4onFFlMWb6BAeJ8G51yLFqlfupmdketAmrNJizewb++OtPeZ8ZxzLViDCYak\n/4Q/yyWVhT9Tr7KGjt1dxMMJTUYP8OIo51zLlm1O7yPDn96sth6friynqibBwY0wRpRzzjVlkYqk\nJA2W1Dp8f6yka8LZ8nZ7kxavB9hq7HrnnGuJoo6t+yyQkLQ3wfzZ/YAnchZVMzJ58QZ6d2pDnwxT\nmTrnXEsSNcFImlkcOBP4g5ldD/TOXVjNw/LSTbz32RrGDPLiKOdcyxc1wYhJugD4FvBSuK5VbkJq\nHuKJJNc+OZVk0vjhCUPzHY5zzuVc1ATjEmAs8CszWxhO2fq33IXV9N3z1jwmLd7Ar886gIFp02o6\n51xLFXV480+AawAkdQFKzOz2XAbWlP1n3lrufWc+5xzclzNG9sl3OM45t0tEbSX1rqSOkroCU4C/\nSNothzZfvK6Sq56YwpCeHbjljNxOkOScc01J1CKpTmZWBpwFPGpmhwIn5C6spqmiOs53Hp2EBA9c\nfAjtir1nt3Nu9xE1wSiS1Bs4ly2V3rudG5+dwedrKrnvGwfRv1u77Ac451wLEjXB+D/gdWC+mU2U\ntBcwL3dhNT1lm2O8MnMllx4xkCP27p7vcJxzbpeLWun9NPB02vIC4OxcBdUUTVy4nqTBccN65jsU\n55zLi0gJhqQ2wGXAfkCb1HozuzRHcTU54z9fR3FRAQf19yFAnHO7p6hFUn8D9gC+DLwH9AXKcxVU\nUzR+wTpG9etMm1aF+Q7FOefyImqCsbeZ/RyoNLNHgNOAQ3MXVtNSWlXDJyvLGDu4W75Dcc65vIk8\nNEj4s1TS/kAnYLcpzJ+wcD1mMHYvTzCcc7uvqB0JxoU9vH8OvAh0AH6Rs6iamPEL1tG6qICR/X1E\nd+fc7itqK6kHwrfvAXvlLpymafzn6xg9sAuti7z+wjm3+2owwZD0o4a2m1mLHx5kfWUNc74o57qT\nfERa59zuLVsOY7edmvWX/5zF1KWlVFbHAbzC2zm328s2p/ctuyqQpua5Kcvp1K4VQ3p2YPSArhzY\n1+svnHO7t6ij1T6SPoe3pC6SHoxw3MmS5kqaL+nGDNu7SHpe0gxJE8IWWEjaR9K0tFeZpB9sz4Xt\nrJpEktMO6M1Dl4zh9q+PoKgwaoMy55xrmaK2khphZqWpBTPbIGlUQwdIKgTuA04ElgETJb0Yzq2R\n8hNgmpmdKWlYuP/xZjYXGJl2nuXA81EvqjHEEklaeSLhnHO1ot4RC8JmtQCE82JkS2zGEAxWuMDM\naoCngDPq7DMceBvAzOYAAyX1qrPP8cDnZrY4Yqw7LZE0koYnGM45lyZqDuMuYLyk1ACE5wC/ynJM\nH2Bp2vIytu0dPp1gjo0PJI0BBhAMO7IqbZ/zgSfr+xBJVwBXAPTv3z9LSNHEEkkAWhWpUc7nnHMt\nQaRHaDN7lODGvip8nWVmjTGn921AZ0nTgO8DU4FEaqOkYuB00kbKzRDbODMbbWaje/To0QghpSUY\nBZ7DcM65lMhTxoV1D59k3XGL5UC/tOW+4br0c5YBlwBIErAQWJC2yynAFDNLz3HkXCxhALQq9ByG\nc86l5PIReiIwRNKgMKdwPsGwIrUkdQ63AVwOvB8mIikX0EBxVK5sKZLyHIZzzqXkbFJqM4tLuppg\npr5C4EEzmy3pynD7/cC+wCOSDJhNMOcGAJLaE7Sw+m6uYqxPTTxMMLzS2znnakWdQOl2M/txtnV1\nmdkrwCt11t2f9n48kHHMDTOrBPLSvTqeDIqkij3BcM65WlHviCdmWHdKYwbSlNQWSXmC4ZxztbIN\nPvg94H+AwZJmpG0qAf6by8DyKVUkVeSV3s45VytbkdQTwKvAb4D0oT3KzWx9zqLKs1QOw4uknHNu\niwbviGa20cwWAfcA681scdjjOi6pxU7RuqVZrScYzjmXEvWO+CegIm25IlzXIm2pw/AiKeecS4ma\nYMjMLLVgZkly2CQ337wfhnPObSvqHXGBpGsktQpf17J1j+wWJVUk5XUYzjm3RdQ74pXA4QRDe6QG\nEbwiV0HlWyqH4a2knHNui0jFSma2mmBoj92C98NwzrltRZ1xb6iktyTNCpdHSPpZbkPLn1Q/DC+S\ncs65LaLeEf8C3ATEAMxsBi04x+HNap1zbltR74jtzGxCnXXxxg6mqYgnvVmtc87VFTXBWCtpMGAA\nkr4OrMxZVHlWO1qtN6t1zrlaUftSXAWMA4ZJWk4w0dGFOYsqz7xZrXPObStrgiGpABhtZieEc1QU\nmFl57kPLn9pmtQVeJOWccylZH6HDXt03hO8rW3piAUGCIUGhJxjOOVcrapnLm5Kuk9RPUtfUK6eR\n5VFNIkmrwgKCacadc85B9DqM88KfV6WtM2Cvxg2naYgnzOsvnHOujqh1GBeZ2Ye7IJ4mIZZIepNa\n55yrI2odxr27IJYmIxYWSTnnnNsi6l3xLUlnazcp1K+JmycYzjlXR9S74neBp4EaSWWSyiWV5TCu\nvPIiKeec21bU0WpLch1IU+JFUs45t63Is+ZJOh04Olx818xeyk1I+ecJhnPObSvq8Oa3AdcCn4Sv\nayX9JpeB5VMsYT6OlHPO1RE1h3EqMDJsMYWkR4CpBEOetzixRJJir8NwzrmtbM9jdOe0950aO5Cm\nJJZIUlTgOQznnEsXNYfxG2CqpHcAEdRl3JizqPKsJmG0LfYEwznn0kVtJfWkpHeBQ8JVPzazL3IW\nVZ7F4l4k5ZxzdUWt9D4TqDKzF83sRWCzpK/lNrT88VZSzjm3rah3xV+a2cbUgpmVAr/MTUj5F096\nT2/nnKsr6l0x036R+3A0NzVxz2E451xdUe+KkyTdLWlw+LobmJzLwPLJhwZxzrltRU0wvg/UAH8H\nngI2s/XcGC2K12E459y2oraSqmQHmtFKOhm4BygEHjCz2+ps7wI8CAwmSIQuNbNZ4bbOwAPA/gST\nNV1qZuO3N4YdEUt4HYZzztWVs7uipELgPuAUYDhwgaThdXb7CTDNzEYAFxMkLin3AK+Z2TDgQODT\nXMVaV00iSasiL5Jyzrl0uXyMHgPMN7MFZlZDUJR1Rp19hgNvA5jZHGCgpF6SOhF0DvxruK0mbJm1\nS8QTSZ+i1Tnn6mjwrijp9vDnOTtw7j7A0rTlZeG6dNOBs8LPGAMMAPoCg4A1wEOSpkp6QFL7HYhh\nuyWSRtLwIinnnKsj213x1HCWvVwNMngb0FnSNIKK9alAgqBu5SDgT2Y2Cqi3DkXSFZImSZq0Zs2a\nnQ4olkhUeAW0AAAY6UlEQVQCnmA451xd2Sq9XwM2AB3CGfZEUAEtwMysYwPHLgf6pS33DdfVMrMy\n4BKAMGFaCCwA2gHLzOzjcNdnqCfBMLNxwDiA0aNHW5bryaqmNsHwOgznnEvX4GO0mV1vZp2Bl82s\no5mVpP/Mcu6JwBBJgyQVA+cDL6bvIKlzuA3gcuB9MysLx6laKmmfcNvxBPNw5Fws7jkM55zLJGqz\n2jMk9WLL4IMfm1mD5T9mFpd0NfA6QbPaB81stqQrw+33A/sCj0gyYDZwWdopvg88HiYoCwhzIrkW\nSwSZFE8wnHNua5ESjLDS+07gXYLiqD9Iut7MnmnoODN7BXilzrr7096PB4bWc+w0YHSU+BpTzIuk\nnHMuo6jjQf0MOMTMVgNI6gG8SVC30KKkEoxin6LVOee2EnnwwVRiEVq3Hcc2K14k5ZxzmUXNYbwm\n6XXgyXD5POoUNbUUqRxGUYEXSTnnXLqold7XSzoLODJcNc7Mns9dWPlT26zWi6Scc24rkee0MLPn\ngOdyGEuTkGpW60ODOOfc1vyuWEc86XUYzjmXid8V6/Ce3s45l1nkIqmwA90wgqFB5oYj0LY43tPb\nOecyi9px7zTgfuBzgo57gyR918xezWVw+eDNap1zLrOoOYy7gOPMbD6ApMHAy0ALTDC8SMo55zKJ\n+hhdnkosQguA8hzEk3c1Pry5c85l1GAOI+x7ATBJ0ivAPwjqMM4hGI22xfGhQZxzLrNsRVJfTXu/\nCjgmfL8GaJuTiPIs7nUYzjmXUYMJhpntkiHFmxKvw3DOucyitpLqAXwHGJh+jJldmpuw8sfrMJxz\nLrOoraT+CXxAMKR5Infh5F8s7kVSzjmXSdQEo52Z/TinkTQRsUSSAkGhj1brnHNbifoY/ZKkU3Ma\nSRMRSyQ9d+GccxlEvTNeS5BobJJUJqlcUlkuA8uXWMJ8pFrnnMsg6nwYJbkOpKmIJZI+F4ZzzmXQ\n4J1R0sAs2yWpb2MGlG9BkZTXXzjnXF3Zchh3SCogaCU1maDDXhtgb+A44Hjgl8CyXAa5K9UkkhQV\neA7DOefqytZx7xxJw4ELgUuB3kAV8CnBnN6/MrPNOY9yF4olzIcFcc65DLLWYZjZJ8BPd0EsTUIs\n7kVSzjmXiT9K1xFPerNa55zLxO+MddQkzBMM55zLwO+MdcTiSe+H4ZxzGUS6M4bNZy+S9Itwub+k\nMbkNLT9iiSRFXofhnHPbiPoo/UdgLHBBuFwO3JeTiPLMhwZxzrnMog4+eKiZHSRpKoCZbZBUnMO4\n8sbrMJxzLrOod8aYpEKC6VlT82MkcxZVHsUTSYqLvEjKOefqippg/B54Hugp6VfAf4Bf5yyqPPIi\nKeecyyzq4IOPS5pMMBSIgK+Z2ac5jSxPYl4k5ZxzGWVNMMKiqNlmNgyYk/uQ8qvGBx90zrmMsj5K\nm1kCmCup//aeXNLJkuZKmi/pxgzbu0h6XtIMSRMk7Z+2bZGkmZKmSZq0vZ+9o7xIyjnnMovaSqoL\nMFvSBKAytdLMTq/vgDBnch9wIsFothMlvRiOTZXyE2CamZ0paVi4//Fp248zs7URY2wUwVhSnmA4\n51xdUROMn+/AuccA881sAYCkp4AzgPQEYzhwG4CZzZE0UFIvM1u1A5/XKGJJr8NwzrlMIt0Zzew9\ngvqLkvD1abiuIX2ApWnLy8J16aYDZwGEPccHAKkJmQx4U9JkSVdEiXNnmRmxRJJir8NwzrltRB0a\n5FxgAnAOcC7wsaSvN8Ln3wZ0ljQN+D4wFUiE2440s5HAKcBVko6uJ7YrJE2SNGnNmjU7FUwiaZjh\nOQznnMsgapHUT4FDzGw11HbcexN4poFjlgP90pb7hutqmVkZcEl4TgELgQXhtuXhz9WSnico4nq/\n7oeY2ThgHMDo0aMt4vVkFEsEhxd5guGcc9uIemcsSCUWoXURjp0IDJE0KBxG5HzgxfQdJHVOG2Lk\ncuB9MyuT1F5SSbhPe+AkYFbEWHdYTSLovO7Nap1zbltRcxivSXodeDJcPg94taEDzCwu6WrgdaAQ\neNDMZku6Mtx+P7Av8IgkA2YDl4WH9wKeDzIdFAFPmNlr0S9rx8TCBMOnaHXOuW1F7el9vaSzgCPD\nVePM7PkIx71CMPd3+rr7096PB4ZmOG4BcGCU2BpTrDaH4QmGc87VFSnBkDQIeMXMnguX20oaaGaL\nchncrhYP6zA8wXDOuW1FvTM+zdaj0ybCdS2K12E451z9oiYYRWZWk1oI37e4+TBq6zA8h+Gcc9uI\nemdcI6l2GBBJZwC7dMiOXSEW92a1zjlXn6itpK4EHpd0L8Hw5kuBi3MWVZ54kZRzztUvaiupz4HD\nJHUIlytyGlWeeJGUc87VL+rQINdK6kgwUu3vJE2RdFJuQ9v1altJeT8M55zbRtQ746XhMB4nAd2A\nbxKOMtuSeD8M55yrX9Q7Y6pQ/1TgUTObnbauxfA6DOecq1/UBGOypDcIEozXw3GeklmOaXY8h+Gc\nc/WL2krqMmAksMDMqiR1IxxltiXxBMM55+oXtZVUEpiStryOYMTaFiXVD8OLpJxzblv+KJ0mlvRm\ntc45Vx+/M6aJxb1Iyjnn6hO1DgNJhQTzVNQeY2ZLchFUvmyO+3wYzjlXn6jDm38f+CWwii2towwY\nkaO48qKyOo4E7YoL8x2Kc841OVFzGNcC+4SV3S1WRXWc9sVFhDP9OeecSxO17GUpsDGXgTQFldVx\nOrSOXErnnHO7lah3xwXAu5JeBqpTK83s7pxElSeV1Qnat/biKOecyyRqgrEkfBXTAidOSin3HIZz\nztUrase9WwBa+vDmldVx2nuC4ZxzGUUd3nx/SVOB2cBsSZMl7Zfb0HY9TzCcc65+USu9xwE/MrMB\nZjYA+F/gL7kLKz8qquOUeILhnHMZRU0w2pvZO6kFM3sXaJ+TiPKownMYzjlXr8itpCT9HPhbuHwR\nQcupFsWLpJxzrn6RZ9wDegDPha8e4boWozqeIJYwOnizWuecyyhqK6kNwDU5jiWvKqsTAN6s1jnn\n6tHg3VHS78zsB5L+RTB21FbM7PScRbaLVVbHAbxIyjnn6pHt7piqs7gz14HkW/nmIMHwHIZzzmXW\n4N3RzCaHb0ea2T3p2yRdC7yXq8B2tcoaz2E451xDolZ6fyvDum83Yhx5VxEWSXVo4wmGc85lkq0O\n4wLgG8AgSS+mbSoB1ucysF0tVYfhRVLOOZdZtrvjf4GVQHfgrrT15cCMXAWVDxWbvUjKOecakq0O\nYzGwGBi7a8LJn9oiqWJPMJxzLpOogw8eJmmipApJNZISksoiHHeypLmS5ku6McP2LpKelzRD0gRJ\n+9fZXihpqqSXol/Sjkn1w/D5MJxzLrOold73AhcA84C2wOXAfQ0dIKkw3OcUYDhwgaThdXb7CTDN\nzEYAFwP31Nl+LfBpxBh3SmVNnDatCigqjPqVOOfc7iXy3dHM5gOFZpYws4eAk7McMgaYb2YLzKwG\neAo4o84+w4G3w/PPAQZK6gUgqS9wGvBA1Bh3RvlmnzzJOecaEjXBqJJUDEyT9FtJP4xwbB+CucBT\nloXr0k0HzgKQNAYYAPQNt/0OuAFIRoxxp/jAg84517CoCcY3gULgaqAS6Aec3QiffxvQWdI04PvA\nVCAh6SvA6rSOg/WSdIWkSZImrVmzZocDqayO094rvJ1zrl5RBx9cHL7dBNwS8dzLCRKWlL7huvTz\nlgGXAEgSsJBg2PTzgNMlnQq0ATpKeszMLsoQ2ziCCZ4YPXr0NuNdRVVRHfdOe84514BsHfdmkmHQ\nwZSwsro+E4EhkgYRJBTnE3QCTD9/Z6AqrOO4HHg/TERuCl9IOha4LlNi0ZgqquP06tgmlx/hnHPN\nWrZH6q+EP68Kf6ZPoNTg07yZxSVdDbxOUJz1oJnNlnRluP1+YF/gEUlGMF/4Zdt/CY3D6zCcc65h\nUTruIelEMxuVtunHkqYA2/StqHP8K8Arddbdn/Z+PDA0yzneBd5taJ/GUFGd8MmTnHOuAVErvSXp\niLSFw7fj2Gahstqb1TrnXEOi3iEvAx6U1AkQsIEWNEVrPJFkUyzhRVLOOdeAqK2kJgMHhgkGZrYx\np1HtYpU1Pj2rc85lk62V1EVm9pikH9VZD4CZ3Z3D2HYZn57VOeeyy3aHbB/+LMl1IPnkc2E451x2\n2VpJ/Tn8GbWzXrNU7gmGc85lla1I6vcNbTezaxo3nPzwIinnnMsu2x0y61hOLcGWBMP7YTjnXH2y\nFUk9sqsCyaeKcPKkktat8hyJc841XZHKYCT1AH5MMH9F7YBLZvalHMW1S1VsjgGew3DOuYZE7a39\nOMHMd4MIRqtdRDC4YIuQ6ofhdRjOOVe/qAlGNzP7KxAzs/fM7FKgReQuIBiptqhAtC5qUaOdOOdc\no4r6SB0Lf66UdBqwAuiam5B2vcpwLoxUh0TnnHPbippg3BoOC/K/wB+AjsAPcxbVLlax2Wfbc865\nbKLeJT8Ox4/aCByXw3jyosJHqnXOuayiFtp/KOkNSZdJ6pLTiPKgsibuLaSccy6LSAmGmQ0Ffgbs\nB0yW9JKknE6ZuitVVPvQ5s45l03kZkFmNsHMfgSMAdYDLaZTX2V1nJI2nmA451xDIiUYkjpK+pak\nV4H/AisJEo4WwSu9nXMuu6h3yenAC8D/hfNwtyiV1XEvknLOuSyi3iX3MjPLaSR5dPy+PRnRt1O+\nw3DOuSYt6hStLTaxAPjd+aPyHYJzzjV5PhaGc865SDzBcM45F0nUVlK/DVtKtZL0lqQ1LakfhnPO\nueyi5jBOMrMy4CsEQ5vvDVyfq6Ccc841PVETjFTl+GnA0+G4Us4553YjUZvVviRpDrAJ+F44A9/m\n3IXlnHOuqYk6ltSNwOHAaDOLAZXAGbkMzDnnXNMStdL7HILZ9hKSfgY8BuyZ08icc841KYrSJ0/S\nDDMbIelI4FbgDuAXZnZorgPcHpLWAIu345DuwNochbOrtZRr8etoelrKtfh1ZDbAzHpE2TFqHUYi\n/HkaMM7MXpZ06w6FlkNRLzpF0iQzG52reHallnItfh1NT0u5Fr+OnRe1ldRySX8GzgNekdR6O451\nzjnXAkS96Z8LvA582cxKga54PwznnNutRG0lVQV8DnxZ0tVATzN7I6eR7Rrj8h1AI2op1+LX0fS0\nlGvx69hJUSu9rwW+AzwXrjqToC7jDzmMzTnnXBMSuZUUMNbMKsPl9sB4MxuR4/icc841EVHrMMSW\nllKE79X44ew6kk6WNFfSfEk35jueqCT1k/SOpE8kzQ5zf0jqKunfkuaFP7vkO9YoJBVKmirppXC5\nuV5HZ0nPSJoj6VNJY5vjtUj6Yfh3NUvSk5LaNIfrkPSgpNWSZqWtqzduSTeF//tzJX05P1FnVs+1\n3BH+bc2Q9Lykzmnbdtm1RE0wHgI+lnSzpJuBj4C/5iyqHJNUCNwHnAIMBy6QNDy/UUUWB/7XzIYD\nhwFXhbHfCLxlZkOAt8Ll5uBa4NO05eZ6HfcAr5nZMOBAgmtqVtciqQ9wDcGIDvsDhcD5NI/reBg4\nuc66jHGH/y/nA/uFx/wxvCc0FQ+z7bX8G9g/LNX5DLgJ8nAtZhbpBRxE8Md0DTAq6nFN8QWMBV5P\nW74JuCnfce3gtfwTOBGYC/QO1/UG5uY7tgix9yX4R/4S8FK4rjleRydgIWERb9r6ZnUtQB9gKUEr\nyCLgJeCk5nIdwEBgVrbvv+7/O0EL0LH5jr+ha6mz7Uzg8XxcS9aOe2FqNduCJ6cp2fZvJlL/GCnL\ngCbVaz0KSQOBUcDHQC8zWxlu+gLolaewtsfvgBuAkrR1zfE6BgFrgIckHQhMJsg5NatrMbPlku4E\nlhAMNPqGmb0hqVldR5r64u5DUEqSsixc11xcCvw9fL9LryVrkZSZJYC5kvrnKgi3/SR1AJ4FfmDB\nXCW1LHjUaNLzsEv6CrDazCbXt09zuI5QEUEO/E9mNopgcM6tim2aw7WEZfxnECSAewLt606U1hyu\nI5PmGnddkn5KUCz9eD4+P+rQIF2A2ZImEPwzAGBmp+ckqtxbDvRLW+4brmsWJLUiSCweN7NUU+dV\nknqb2UpJvYHV+YswkiOA0yWdCrQBOkp6jOZ3HRA81S0zs4/D5WcIEozmdi0nAAvNbA2ApOcIRqlu\nbteRUl/czfL/X9K3CSaxOz5MAGEXX0vUSu+fEwT6f8Bdaa/maiIwRNIgScUElUYv5jmmSCSJoMHB\np2Z2d9qmF4Fvhe+/RVC30WSZ2U1m1tfMBhJ8/2+b2UU0s+sAMLMvgKWS9glXHQ98QvO7liXAYZLa\nhX9nxxNU3je360ipL+4XgfMltZY0CBgCTMhDfJFJOpmg+PZ0CzpSp+zaa8lS8bI3cESG9UcCg/Nd\nMbQzL+BUgtYGnwM/zXc82xH3kQRZ6xnAtPB1KtCNoAJ5HvAm0DXfsW7HNR3LlkrvZnkdwEhgUvh7\neYEgV97srgW4BZgDzAL+BrRuDtcBPAmsBGIEOb7LGoob+Gn4vz8XOCXf8Ue4lvkE9a6p//n783Et\nDXbcC9vG32RmM+usPwD4tZl9td6DnXPOtSjZiqR61U0sAMJ1A3MSkXPOuSYpW4LRuYFtbRszEOec\nc01btgRjkqTv1F0p6XKCtubOOed2E9nqMHoBzwM1bEkgRgPFwJkWtA5xzjm3G4g6Wu1xwP7h4mwz\nezunUTnnnGtyok6g9I6Z/SF8eWLh6iXJJN2VtnxdOGBlY5z7YUlfb4xzZfmcc8IRZ9/JsO2OcDTX\nO3bgvCPDjopNlqSKHTzuazsygOeOfp7LD5+X2zW2auAsSd3zHUg6SVFHNYCg3ft3zOy4DNuuAEaY\n2Y5MUTySoM9MZAo0h//TrxGM/OxasObwh+ialzjBFJI/rLuhbg4h9XQp6VhJ70n6p6QFkm6TdKGk\nCZJmShqcdpoTJE2S9Fk4HlVqTo07JE0M5wv4btp5P5D0IkHP67rxXBCef5ak28N1vyDoHPnXurmI\n8DwdgMmSzpPUQ9Kz4edOlHREuN8YSeMVzPPxX0n7hCMK/B9wnqRp4fE3S7ou7fyzJA0MX3MlPUrQ\nga6fpJPCc06R9HQ4lhjhd/VJeN13ZrjGY8LPmxbGUxKuvz7t+7ol0y+yvn0kXRyumy7pb5IOB04H\n7gg/Z3D4ek3S5PB3MCw8dlB4HTMl3Zrpc10Tlu9ejf5qWS+gAugILCIY9vs64OZw28PA19P3DX8e\nC5QSDEHdmmAsnFvCbdcCv0s7/jWCB50hBL1g2xA89f8s3Kc1QY/rQeF5K4FBGeLck2AojB4EY6q9\nDXwt3PYuwZwQGa8v7f0TwJHh+/4Ew7UQXn9R+P4E4Nnw/beBe9OOvxm4Lm15FkH/poFAEjgsXN8d\neB9oHy7/GPgFQU/muWypi+ycId5/EY7WQJDYFREMWT6OYBK0AoJhzI+u8zvJuA/BvAufAd3D/brW\n87t9CxgSvj+UYOgXCIayuDh8f1X69+mvpv/anmy6c5GYWVn4dHwNwTDZUUy0cChqSZ8Db4TrZwLp\nRUP/MLMkME/SAmAYwc1tRFrupRNBglIDTDCzhRk+7xDgXdsy0N7jBDfEFyLGC0FiMFyqnXyyY/jk\n3wl4RNIQgmFcWm3HOVMWm1lq2OrDCIp7Pgw/qxgYD2wENhPkhl4iuKnX9SFwd3h9z5nZMkknEXxn\nU8N9OhB8X++nHVffPgcCT5vZWgAzW1/3A8Pv4HDg6bTvpnX48wjg7PD934Dbs34TrsnwBMPlyu8I\n5k95KG1dnLAYNCyXL07bVp32Ppm2nGTrv9O6zfqM4Cn4+2b2evoGSceSNrpyDhQQ5AI21/nce4F3\nzOxMBXOWvFvP8bXfR6hN2vv0uAX828wuqHsCSWMIBgn8OnA1wWRUtczsNkkvE9SdfKhgCk8BvzGz\nPzdwbRn3kfT9Bo5JKQBKzWxkPdub/TDjuyuvw3A5ET55/oOgAjllEXBw+P50duzJ+xxJBWG9xl4E\nRTKvA99TMOw7koZKap/lPBOAYyR1VzBJ2AXAe9sZyxtA7Q1UUuoG2YktQ0x/O23/craeLGoRwTwa\nSDqIoBgtk4+AIyTtHe7bPrzGDkAnM3uFoM7owLoHShpsZjPN7HaCUZqHEXxfl6bVg/SR1LPOofXt\n8zbB76BbuL5r3WuzYH6WhZLOCfeRgomlIMjxnB++v7Ce63VNlCcYLpfuIih/T/kLwU16OsE0uTvy\n9L+E4Gb/KnBl+HT/AEGl9hRJs4A/kyX3HBZ/3Qi8A0wHJpvZ9g7bfQ0wOqwA/gS4Mlz/W+A3kqbW\nieMdgiKsaZLOI5jTpKuk2QS5g8/qiXUNQcLzpKQZBMVRwwhu0C+F6/4D/CjD4T8IK9NnEIx++qqZ\nvUFQ/zJe0kyC+TvSEzLq28fMZgO/At4Lf4+pIfafAq4PK9YHEyQGl4X7zCaYmAmCOqmrwnM2p1nu\nHBE77jnnnHOew3DOOReJJxjOOeci8QTDOedcJJ5gOOeci8QTDOecc5F4guGccy4STzCcc85F4gmG\nc865SP5/g269aSSgxqgAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x9c84471cf8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "rfecv_R2L = RFECV(estimator=clf_R2L, step=1, cv=10, scoring='accuracy')\n",
    "rfecv_R2L.fit(X_R2L_test, Y_R2L_test)\n",
    "# Plot number of features VS. cross-validation scores\n",
    "plt.figure()\n",
    "plt.xlabel(\"Number of features selected\")\n",
    "plt.ylabel(\"Cross validation score (nb of correct classifications)\")\n",
    "plt.title('RFECV R2L')\n",
    "plt.plot(range(1, len(rfecv_R2L.grid_scores_) + 1), rfecv_R2L.grid_scores_)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZIAAAEbCAYAAADwPQLqAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzsnXl8nHWd+N+fyX2fbdqkbRIKvShtCm1BFvBEucuxXCoq\nosguKuguXqvr6upPwROPXRaFXVQulVPAgiK3QFva0DMtpU2PNGmbJmnuY2Y+vz+e55nMJJPkaek0\nafJ5v17zysxzzWeezHw/38/5FVXFMAzDMA6XwGgLYBiGYRzbmCIxDMMw3hGmSAzDMIx3hCkSwzAM\n4x1hisQwDMN4R5giMQzDMN4RpkgMwzCMd0Syn4NEZDLwD0Ap0AWsB1apajiBshmGYRjHADJcQaKI\nvBf4ClAIrAH2AenALGAm8EfgR6ramnhRDcMwjLHISIrkB8DPVXVnnH3JwAVAkqo+lDgRDcMwjLHM\nsIrEMAzDMEbCV7BdRG4SkVxxuEtEVovIBxMtnGEYhjH28Zu19Uk3DvJBoAC4Bvh+wqQyjKOEiNSK\nSJeItItIg4j8n4hkR+3/PxHpdfd7jyvjnOs9fhF17lR34lUvIm0iUiMi3xKRLPf5J+PIc5OIrBpC\nVhWR4wds+w8R+Z37/DQR+YuINInIfhH5g4hMHeKzNLnHznnnd9GY6PhVJOL+PQ/4rapuiNpmGMc6\nF6pqNlAFLAK+OmD/baqaHfV4cOC5UY/PAohIIfAqkAG8S1VzgLOBPJxElXuAj8WR5Rp33+FQANwJ\nVADlQBvwv/E+C1AG1AF3HeZ7GUYEX+m/wBsi8gxQCXxVRHIAS/01xhWq2iAiT+MolHfKF3EG8o96\nafKqugu4GUBEmoD/FJFyVd3hbpsHLADuP0z5/xz92rWOXhji2C4R+T3wh8N5L8OIxq9Fch1OGvAS\nVe0EUoFrEyaVYYwCIjINOBfYegQu9wHg4aFqrVR1N/AcjgXicQ3wlKo2HoH3BzgL2BBvh4hkAVdz\nZD6rMcHxpUjcH8NeYJ6InAWcCOQnUjDDOIo8KiJtwC6cWqlvDtj/ryLS4j4GDvKPRu1rEZFPu9uL\ngPoR3vceXEUiIgHgIxy+WysGEVkA/Dtwy4Bd/yoiLTjW0hnEKjLDOCz8Zm3dCrwCfB3ni3kL8K8J\nlMswjiYXuzGM9wBzgOIB+3+oqvnuY+C+i6P25avqr9ztB4CpDM/DwFQROc1970zgyWGODwEpA7al\nAH3RG9yA/J+Bm1T1pXifBSeO0gXMHkFGwxgRvzGSi4HZqtqTSGEMYzRR1RdE5P+AH+J8598JfwUu\nEZFvDePe6hSRP+IE3TOAB1S1d5hr7sRRAJuitlUCW7wXIlLuvvd/qupvh7qQqu4UkZuAe0TkCVXt\n8vm5DGMQfmMk2xg8EzKM8chPgbNFZOE7vM6PgVycgbocQETKROTHrtvJ4x7gSuAyRnZrPQh8XUSm\niUhARD4AXIjTqggRKQP+BvxCVe8YSUBV/QuwB7j+0D6aYcTiV5F0AtUi8j8i8jPvkUjBDGM0UNX9\nwG9w4gt++NOAOpJH3Os0AafjuJ1ed2MwzwIHiQ1wv+hu262qK0d4r28DfwdeBpqB24CPqOp6d/+n\ngOOA/4iWaYRr/gD4koik+fy8hjEIXy1SROTj8bar6hEJDBqGYRjHLr57bYlIKk7XX4DNqto33PGG\nYRjGxMCvRfIeHP9tLU5F+3Tg46r6YiKFMwzDMMY+fhXJG8CHVXWz+3oWcL+qnpJg+QzDMIwxjt9g\ne4qnRABUdQuWxWUYhmHgv45klYj8Gvid+/ojQNwOpWOR4uJiraioGG0xDMMwjineeOONRlWdNNJx\nfhXJPwE3Ap93X78E/NdhynbUqaioYNWqY0bvGYZhjAlEZIef43wpErei/cfuwzAMwzAiDKtIROT3\nqnqFiKwDBkXlVXVBnNMMwzCMCcRIFslN7t8LEi2IYRiGcWwybNaWqnptsP9ZVXdEP4B/Trx4hmEY\nxljHb/rv2XG2nXskBTEMwzCOTUaKkfwTjuVxnIisjdqVg7M+iWEYhjHBGSlGch/OAjnfw1lq16PN\n7W5qGIZhTHBGipEcVNVaVb3ajYt04WRvZYvIjKMioWEY45JgKMyDK3cSCvtrHGuMXfwutXuhiLwF\nbAdewGne+OcEymUYxjjn1W0H+PJD63hla+Noi2K8Q/wG278DnAZsUdVK4P3AawmTyjCMcU9Lp7MS\nxe5mW+X3WMevIulT1QNAQEQCqvocsDiBchmGMc5p6w4CsKfFFMmxjl9F0iIi2TjLgt4rIrcDHYkT\nyzCM8cTmhjbOuu059rZ2R7a1dTsWydFSJPf8vZZr7nr9iF3vmQ0NnHv7S3T1ho7YNY9V/CqSZTjr\ntn8BWA68DVyYKKEMwxhfPL2hgZ1NnWzd17+EvGeR7D5KiuSVrY289FYjDQe7Rz7YB2/sbGZTfSt/\n3bT3iFzvWMavIpkMpKpq0F2n/Vc4tSSGYRgjsrLWqRZo7epfoftoWyR7Djrvs6L2yFQuNLb1AvDo\nmrojcr1jGb+K5A9AOOp1yN1mHEFW1jaZv9g4JJ7bvI+DnX0jH+iTV98+cMS/g8FQmNU7mgE4GKVI\nWl2LpOFg91FJAd7T4lgiK7cfIUXS3gPAC1v209TRO+Rx7T1Blq9vOKz36A2GeWpdPX5Wsh1N/CqS\nZFWN3Cn3eWpiRJq4XP+bVfz8b1tHWwzjGGF3cyfX/u9Kfve6ryUjRqSls5eP3f06v3zuyH4HN9W3\n0eHGEVq7B1skwbCyr+3IuJuGorM3GBnsVx4hi+RARw9l+RkEw8qT6+qHPO6ev9dyw+/eoO4wFPSf\n19fzz/eu5rVtY7v+268i2S8iF3kvRGQZYMnfR5DuvhDNnX3UHzSLxPCHNyBu239k8l6eXFdPX0jZ\ndYTTcaNdSQMtEhHneaItcc8aOW5SFpv3th0RK66xrZd3zSxiVkk2jw3j3lrhWkC7mzoP+T027mmN\nucZYxa8iuQH4mojsFJFdwJeBzyROrInHAXe2tLe1Z5QlMY4VVmx33EW1B46MInlszR7gyA/qK7c3\nMa0gg4LMlBhF0tYdZEZhJpD4WhLPGrikqgxVWLXjnQ3MqsqBjh6Ks9NYVlXGqh3N7IqjKEJhjbj1\nDsci2dTQBhw5KypR+FIkqvq2qp4GzAPmqurpqmo+mCPIAdffuq81sSb+0eDe13dw8wNrRluMcY83\nuNQ2+lckf9m4l8Xf+QuLvv0Mp/znX/jNq7WA4yZbUdtERkoSdc1dvn3yL27Zz7m3vzTkDF9VWVnb\nxNKKQnIzUmjtCkb2tXX3MbvEydnxLIZ4bNnbxod+8mJkdu6H3722gy/+vjry2lOO5540lZQkYWVt\ns+9rAfx1416uuev1yH052NVHX0gpzk5lWVUpAPe+vnPQeTUNrbT1HH69zOYG5zOv3tlMMBQe4ejR\nY1hFIiIfdf9+UUS+CFwPXB/12jhCeIG7Ax299AbH7hfGD0+ureeJtfX0jeEv/rFOU0cvW/e1U5SV\nyoGO3pjYw3A8u2kv3X1hLlxYSkVxFt/+00aqd7Xw+JuONXLV0ul09YUiVefDsb+thy88WM2m+lbW\n7Io/MG9v7OBARy9LKgvJy4i1SFq7+pial05eRgp1LfHdPt19IT5//xo2723j3kOIBf3pzT08Vr2H\n7j4nNrOnpYukgFBRlMn8srxDnuG/7KYON7Y7ngPv9zopJ41pBZlcsqiMX720jdU7Y++DF9hPTQ5Q\nN4yyjEdTRy97W3tYOC2Pzt4QGw5BkR5tRrJIMt2/OUM8jCOEl0oIsL/92HVvqSo1DW0Ew0qdtb5I\nGKvcgfCSRWUA7Gj053/f1NDGSWV5fHvZfO7+xBJKctO5+YE1PPTGbk4pL+DUyiJgZDeMqnLLH9+M\nzLZrXBfMQLwBe0mFo0g8haeqtPcEyUlPoSw/Y0iL5Lblm6lpaOP4ydk8ua7e1yTL+w6GwhqpW6lr\n7mJKbjrJSQGWVhSydndLRMn4wSuk9KwKT6EUZ6cB8K1lJzI1L52bH6imvaff6lpZ20xpXjpzpuQc\nsmurxrVGPnpauXutseveGkmRzHT/blTVbw18JFq4iUS08tjr073VFwrz4pb9iRLpsNjf3hPJjjlS\nvvt3QmdvcNA96g2GeemtsXXfDpWVtU2kJge4yHWrxLvXBzv7eCMqFhAOK1sa2pgz1ZkD5mWk8OMr\nFrKjqZO393dwcVUpZfkZwMiK5J6/1/L85v18/fy5TM1LZ/MQimTF9mYKs1KZOSmL3PR+i6SjN0RY\nISc9mdL8jLhunxe27OfuV7bzidMr+Np5c2jp7PP1fd/b2hN5H0+uupYuSvPTAUep9YWU6l0tcc9v\n6ewdZFl4v8m6iCJxfq+eIslNT+GnV1axu7mTbz62Aeh36y2pLHSV5aEpEk/2d8+exIzCzEjAvS8U\n5u9bG8dU1+SRFMl5IiLAV4+GMBOZxihF4jdO8ttXd/Cxu1ewYc/BRIl1yNTU9w8oh+K7TwSqyi1/\nWMvH7l7B1n39cv1+1S6uuWsFW/bGH/yOBVbUNrNwWh4nTHaUQrx7/f3lm7jif16LDKo7mzrp6gsx\nd0pu5JhTjyvic+87gey0ZM5fUEpZgaNIhhv0VJWf/20rZxxfzDWnlTN7Sg6b6ge7XXYe6GT5+npO\nn1mEiLgxEkcWL/XXsUjS41qvP3/2LSqLs/jKuXM484RJFGal8mj1yMV/mxr6ZfFm9XsOdkWU5Cnl\nBQCDlIXHd57cxNV3vhYTk/CSYCIWSZunSPqrIBZXFPLZ953AQ6t386c397CzqZN9bT0sqSikND/j\nkGJP4PyWCrNSmZSdxpKKQlbtaEZVufXPNXz410c+TfudMJIiWQ40AwtEpDXq0SYiIzrsROQcEdks\nIltF5Ctx9heIyCMislZEVojI/Kh9N4nIehHZICI3R21/UESq3UetiFQPvO6xyIH2XnLSnHXG/GZu\neT+qseQ79X64qUkBag8cerrjkeSPb+yO5Pe/HpU+6T0/lODtWKKzN8iGuoMsqSgkIzWJKbnpg+51\nTzDEk2vrY7KGvP/N7CmxXukvnj2Llf/2AQqzUinITCE9JTCsW/Lt/e0c6OjlwoVTERHmTMnl7f3t\nMTGxvlCYmx5cQyAgfPW8uQDkZiTT2hVEVSPtUXIzkikryKCtJxgT5+nuC/Hm7hY+OK+E9JQkUpIC\nnH/SVP6ycW9ECQ2FN5mZUZgZcXHVt3RT6iqSgqzUIa2ort4Qf15XT08wzD5XWaj217nURbm2AgIF\nmbHldJ9/3/EsmpHP1x5Zx6NuFpynSPzGniKfo6GVOVNyEBGWVhbQ1NHL//29ll+/vJ3CrFRuf/Yt\n3thxaEkDiWKkha1uUdV84ElVzY165Khq7nDnikgS8Euctd3nAVeLyLwBh30NqFbVBcDHgNvdc+cD\nnwaWAguBC0TkeFemK1W1SlWrgIeAhw/xM49JGtt7OKEkm5QkocGHRbJtfztrdzuWyFBuhdGgpqGN\nktw0jp+cPaqurdrGDv7j8Q2cWllIcXZaJOipqpHn0TPXY4k1O1sIhpUllYUAlBdlDrrXz9Xsj1SO\ne3Ucm+rbEIFZJYPDmxmpSQCIiOOGGaaeyUs7XlLhvP/cqTn0hTSmnuXnz77Fmp0t/L9LTopYAnkZ\nKfSGwnT3hWMsEm+Aj1Ze1bta6Atp5D0ALl5URk8wzNMbhu9ttbmhldK8dJZWFlLT0Mb+th6CYY28\nD8CcKTlxfzd/3bQ3UjzpWR/NnU6GVrSMje09FGalEQhIzPnJSQF+emUV4bDyk79uIS8jhRMmZ/t2\nGXqEwsqWve0Rpe/dh2/9aSPHT85m+c1nOjGZB9eMqFiPBn7Tf5cdxrWXAltVdZtbCf8ATvPHaOYB\nf3PfowaoEJESYC7wuqp2qmoQZzGtS6NPdF1uVwD3H4ZsY47G9h4m5aQxOSfdV4zk0eo9iEBZfkZk\npjmQW5fX8N/Pvx2z7bblNbz7B8/x7h88x1m39T9uXV4Tc9ydL77NVx9ed8itGWrq25g9JZfK4ix2\njGCRLF9fz2fvWz3iezy7aS/vceV97w+f5+HVu0eU48sPrSUpIPzkyiqWVhZE0j13N3dFFHUiFXBT\nRy9X/M+rvL2/v0nhgfYerrjj1RHft6s3xDV3vc6bQ/jwV9Y2IdLvonHudawieay6juLsVE4qy4sE\n5jc3tFFZlBVRGkPhuWGGYmVtE8XZqVQWZwH9Fo73PVxfd5BfPLeVy06exoULSyPn5WWkAE51u5cG\nnJOeHBlko91pnsyLKwoi206ekc+Mwky+9fgGzrrtOd7/o+fjunVrGtqYPSWHOVNy2N/Ww9rdzn30\n3HYAc6bmsnVf+6Dg/WPVdaQmO8OiN+h7v0eR/n5dje29MW6taMqLsvjWMse5sqSigEBAhlUk3/7T\nRh5cGZs6PNANWVmcRXF2KqlJAX521SIm56Tz0yurqGvu4rblm+PKcTQZKf33ZfdvW5RLy3uMNJ0r\nA3ZFvd7tbovmTVwFISJLgXJgGrAeOFNEikQkEzgPmD7g3DOBvar61ghyHBM0tvdSlJ3G5Nw09o3g\n2lJVHquu4/SZRfzD8UVxB6amjl5+9eI2/uu5rZHslNbuPn798nay05JZND2fk2fkc0p5AWnJAe59\nbQfhqODdb1/bwf0rdvK7OLnxQxEMhdm6r525U3IoL8pkV1PnsLnvD6zcxRNr61kzxIDpceeL2+jo\nDXFKeQHpKUl85eF1QypP77O/vr2JT595HKX5GSypKKSupYs9LV2RgOXcqbkx8ZwjzeodzazY3sTv\nXutPWX1kTR0rapu459XaYc9dv+cgL73VyF0vb4+7f8OeVmZOyiY33RmYy4uyaGzvjcxMW7v7eLZm\nHxcsKOW04wp5c9dBuvtC1DS0DnJrxaMsP2PYVNUV25tYUlGIuGXpxxU7lrSXuXX/ip2kJgf45kWx\nDghP3oNdfRE3Vu4QimRFbTOzS3LIj3IdiQjfvHAe7587mVPKC9jT0h1zf8FJpNi6r505U3OZ4w7C\nz23eF/lcHnOm5BAMK9sa+xV9c0cvz2/ez+WnTAMGK5LZJTkxFsmknLQh79FlJ5fxb+fN5Z/e4+Qr\nDRV76uwN8ptXa/njG7GTo5r6WDekiPCNC+Zx+1VVzCt1PtfiikLeN2cyr28/MKQcR4uRXFtnuH9z\nolxaOX5cWz75PpDvxjk+B6wBQqq6CbgVeAYnTlON0ygymqsZxhoRketFZJWIrNq/f2xn6ARDYZo7\neynOTqPEh0VSvauFHQc6WVZVxpwpuTS297K/LVb5PLl2D8Gw0tYT5G81zg9p+foGeoNhvnPxfH56\n1SJ+etUifnJlFTe8eyat3UG2uAHphoPd7GrqIis1ie88sZG3fAaltzd20BsKM2dqDhXFWU4K8BCm\nfCisvOFaCcN1T93T0sXr25u45rRyfnJlFb+9bim56cncdH/1kOmb3mz2tJlOKqvnFlhZ28TK2iZy\n05NZVlVKQ2s3LZ1DN9t7J3iupj+9WR9Rpo9VOz7zp0ZIY/UGkWc2NsSkkkb2u75zj8piJ0vfswCX\nr3P+zxcvKmNJRSG9oTCvb29iR1NnZHAdjrL8DBrbe+Le3z0tXdS1dMW4nFKTA8yclE1NfSu9wTBP\nrqvng/OmRBSHR8Qi6eqLxEhy0lMozk4jNSkQaSfvxXWWVBYwkPfPLYl8bz90YglPrq2nJ9gv57bG\ndoJhZc6UnEh2mvf9j3VtOfchejLx5Lp6gmHlw6fOID8zJaI0vIndohkFNHf20dkbpLG9J5KxFQ8R\n4dNnHccp5c59Gir25LkpaxraYizzmobBbshlVWWce9LUmPOPm5TNjgOdMZPA0cDvmu0zRSTNff4e\nEfm8iOSPcFodsVbENHdbBFVtVdVr3XjHx4BJwDZ3312qeoqqnoUT8N8SJU8yjiXz4FBvrqp3qupi\nVV08adIkPx9z1Gjq7EUVJmWnUpKbFleR9ARDPLOhgSfX1vM/L2wjNTnAOfOnRAaUgTP0R6v3cMLk\nbCbnpPGIO1A/uqaO8qJMqqbH/usiA607W/fy1X929SKy0pL5/APVPLm2nifX1sesJzEQb0Y6uySX\niiLH7bHdzSZq6eyNUUhexW9OevKwxYteoZxXPVycncYPLl/I5r1tfO3hdTy5tp6n1tXTHNV91UuN\nXTAtD3Csj+y0ZFZsb2JFbROLKwqZOzU3RuZQWCMKyCMcVt7Y0RTzA/dSOkf64XqKpLG9h7+/fYCt\n+9pZV3eQM08opqWzjxeGSWPd5A4i3X1hntkQ2zW2vSfIrqauGEVSPuBeP7KmjoqiTBZOy4v8b3/3\n2g5UBwfa4+ENuPVx1u3wvhtLKwtjtnsxhxe27Kels4+LF5UOOjc3o98iiQTb01MIBISp+emRWpJN\n9a209wRjlFU8li0qo7U7yPOb+++lpxjmTMmlODuN4uxU9rb2kJeRQrabzAJOz61oKwqc38cJk7OZ\nNzU3Jl3X+z0ucn83e1q6XEXiv2+tiDhpzgNiT56F3NYdZE/U/a5paPXlhiwvyqQnGPYVV00kfntt\nPQSE3ID3nTgK4r4RzlkJnCAilSKSClwFPB59gIjku/sAPgW8qKqt7r7J7t8ZOEoj+v0+ANSo6sjO\n8mMArxixODuNybnptHYHB6269rvXdnL9b9/gxvtWs3xDA+fOd2Z83sAQ7d7a1dTJGzuauXhRGRcu\nLOX5zfvY3NDGq9sOcHFVWcQl4TG9MIOS3DRWuBbCytomMlOTePesSdx22QI2N7Ry432rufG+1Xzk\n168Nmb9e09BKckCYOTmLigGz5G8+voGLf/kKHe4M21Na/3L2LJo6enn5rfg9QB9dU8eiGfmRwRLg\nvbMnc90ZlTy8po4b71vNP9+7mm8/sTGyf0VtM1XT8klLdn6ESQHh5PICnt20j237O1hSUchcTwG7\ns/8HVu7kH+94lde39bsJHllTx2X//Sovb+2X7cl19Vx+x6vc+dK2uPJ67DjQyZwpOeSkJ/Pomjoe\nq64jIHDrZQtGTGPd3NDG4vICphVk8KhrxUTvA2Isi/Ii517XNnbw962NvLrtAJeePA0RoSArlRMm\nZ/Osu/jS3Kk+XFvDpACvrG0iOy05RpEBzJ6Sy56D3fzm1VoKs1I584TBk7e8GEXSR3JASE9xhqDj\nirN49e1GDrT3RAbXkRTJmccXU5SVymNR97KmoY2UJOG4Sc73xbtP0dYIQEqSa0W5E7BdTZ2scn8z\n3qAfcW21dVOQmUKle8239rbT3Rce1iKJRzyX4craJtLcmExNVAr1+rrWiEU1HJXu72K0U+39KpKw\nG/S+BPi5qt4CTB3uBPf4zwJPA5uA36vqBhG5QURucA+bC6wXkc042V03RV3iIRHZCPwJuFFVox3p\nVzFOguzQX0NSlJ1GSa5TNDWwrfYja3Yzb2ouz3zhLJ75wlncetmCyDmTctLYFGWiez+sZVWlXFxV\nRl9IuemBNag6mS8DERGWVBSycrsz+16xvYmTZxSQnBTgA/NKePWr7+eZL5zFt5edyN7WHl7bFt8n\nu7mhjeMmZZGWnMSk7DSyUpPY3thBe0+Qpzc00NEb4i8bnQHNq/j98Knl5GemxB1YaxpaqWlo4+Kq\nwTJ//fy5/O1f3s0zXziLZVWlLF/fQEdPsD81doBbZGlFQWTWtqSigEk5aRRkprDZtZIeWV3n3ud+\nObzn3r7o5z98ejPrdg9dv7O9sYPZU3I4b/5Unt7QwMOr6zh9ZjGl+RlcsGAqfx0ijVVV2dzQxpwp\nuVxcVcbLb+2PcVt6A1/0IJOZmkxJbhrVu1r44u/f5LhJWXzqzMrI/iWVhYQVMlOTmF6QyUiUxcmi\n8li5vZlFM/JJToodOjx5XnqrkfNPmkpK0uChJdq11drdR056cmRS8+Vz59DaHeRLf1zLytomyvIz\nBg3+A0lOCnDhwlL+umlfJOZS0+DEj7z39yZaZXGuFZ255Vm+F7nJAWVRdR97W3soyU2PyPOm+38v\nOhxFEnVP+0Jh1uxs4YIFpa7sjiz1Bx33oecWG45yN+FhtFPt/SqSPhG5Gvg48IS7LWWY4wFQ1adU\ndZaqzlTV77rb7lDVO9znr7r7Z6vqparaHHXumao6T1UXquqzA677Ce8a44H+KlnHtQWxtSRb97Wx\nvq6Vy06ZxqySHGaV5JCe0m/yzpmSExlgVJVH1tSxtKKQaQWZzC/LZeakLGoa2lg4LS+SaTOQpZWF\nNLR2s7G+lc1722JmgyW56cwqyeGKxdPJSUuOGWyj2VTfFpkBigjlRVnUHujgmQ0NdPeFSU8J8Gh1\nXUzFb2pygPNOmsozG/ZGrBWPR9fsISkgXLBg8JxFRDhuUjazSnL48NIZdPU5SsrzOS8eMJv1Pk9q\ncoCTpuVF6h821bdFZqMZKUk8tc7xue9t7eaVtxtJTwmwfENDZD2LF7bs58rF0ynOTuOmB9bQ2Ts4\nhtETDLGnpYuKoiyWLSqlozdEXUtXRIkvq3LSWOMtdrS7uYv2niBzpuZw8aJSwgpPrO23Smrq28hJ\nSx40MFYUZfFszT4OdPTws6sWkZna78ZZ6n72WSU5g9JV41GSm47I4Ayjls5eNu9ti1wvmmgLJd5k\nBZwMLYCDXUHaup32KP3n5/KVc+bwbM0+nt7QMMh1NhTLqkrpDYZZvs65lzX1bRG3ZbRcZW5Ve4zM\nU3OpP9jNwc4+Hl1Tx+LyAqa73YjL8jPo6A3R2h1kX2s3k3PTKclJIykgkSywQ3FtgWMVRceeNuxp\npasvxPvmTGZaQUZEkXgWWbz7PJCpuemkJgdGvYuEX0VyLfAu4Luqul1EKoHfJk6sicUBr29PTr9F\nEh0neXTNHgICFy6MbwTOnZrLW/vanZXodrbw9v4Olrk+ahGJzOiXxZnZeyx2Zz//88I2VIkb6ExP\nSeKc+VNYvr5hUCB2f1sPdS1dMTPliuJMdhzo5NHqPUwryOATp1fy0luNrN7Zwr62nshgf3FVGV19\nIZ7Z2D+whsPK49V1nHVC8YgzvyUVhZTmpfNodR0rtsemxnosnJ5PalKAqun9Lq/ZU3LYsrctohi/\nccE8WruFucI5AAAgAElEQVSDPFeznz+9uQdVZ1una0l5CQwfP72CH1+5kO0HOvjQT1/k/J+9xBV3\nvBpZS2ZXUxdhdT7/aZVFTM1LJy05wIdOLAH601i9WXA0NVGuq+Mn53BiaW5MMsJmN7V1oHvSi0nd\n8qHZzC/Li70/lf31Hn5ITQ5QkpM+yLX1ytYDMdeLZkqu03xxRmEmJ8+IHz5NSQqQlZpEa3efq0iS\nY/Zf+w8VvHvWJMIam/Y7HFXT86koyuR7f97E+T97iYbW7hil5imV6NRfD++4h9fs5q197TEKMLq2\nZW9rDyU5aSQnBZiSmx6xRA/HtQVE1oxfGXHhFTBnSm7EtbWytoms1CRf/69AQCgvzDw2XFuqulFV\nP6+q94tIAZCjqrcmWLYJQ2N7D6nJAXLSkinJiVUkqspjb9bxD8cXMzln8KwKnLTE3mCYmoY2vvzQ\nWibnpEXMZYCrT53B1UtncJmb1hj3Gq4//4m1e0hJEhZNj/9DvnhRGe09QZ7dtC+yTVX5t0fWkZIk\nfGBuSWR7RVEWO5s6eWVrI8uqSrlkURmhsPLtPzm9iLwZ1+LyAsryMyKVwOAU0e052D3k7DaaQEC4\nqKqMl95q5JmNe5k7JXdQxlB6ShJfOXcO//yemZFtc6fm0Nkb4v/+XsuSigKuWDyN4mzH5/5odR0L\npuVx9ZIZTM1L57HqPTxavYfZJTnMnZrD6TOLufXSBcwuyWFSThorapsibjuvpqO8KItAQPj3C+bx\n7xfOi8zARYSz55Xw+vammIwj6G8b7rlkLq4q483dB9ne2IGqsmmIFN4rlkznxvfO5FNnHDdoX1l+\nBje9/wSuXjpjxHvpUZqfHmORHOzs47tPbqSyOItFcRSFiHDLh2bz9fPnDlJy0eS6HYDbuvsG/Y9E\nhB9evpCrl07n3PnDes5jzvnKuXM5pbyAKbnpnHPilJhz50zJ4bozKjnnxMHX86znn/9tK8kB4fyo\njChP8exq7mR/ew9T8pzfXml+eqRR5XDpv/EoHVBLsqK2ifKiTCbnOk0dtzV20BMMsaq2mZPLCwa5\nD4fCs/xHE79ZW8+LSK6IFAKrgV+JyI8TK9rEYX97D5Oy09x+RMmkJQci7RlW72xmV1NX3DiBh2cF\nfP6BNWzd186PrlgY8UeDM3P63qUnxWwbSFJAWFxeQFhhflnekNkipx1XREluWox76/4Vu3hm416+\n9KE5MemKFUVZhMJKKKxcXFUWKRJ7c/fBSMUvOIpgWVUpL29tjMQDHquuIzM1ibPnleAHT0ltqm8d\n0i3yyTMqec/syZHX3kDS1NHLsqoykpMCXLCglL9s3Mv6ulaWVZW5SqqUF7bs540dzSxbVBoZKK9Y\nMp1ff3wJ//uJJUzJTY+4JLzsKS8Qeu5JU/nIqeUxsiypKKQ3GB4UZ9nU0Mb0woxIhtFFVaWIOEkH\n9Qe7aesOMmfq4BTeU8oLuOVDc4Z0XX3h7FksmDZSomU/ZQWZEYtEVfnao+vY19bDT6+silh0A/no\naeV88MQpw143L6JIBlsk4AzO37vUSUjwyznzp/Drjy/hrk8s4Y5rTmFGUX8cKDkpwDcumBezzaMk\nN438zBSaOnp5z+xJFES9p9fgcd3ug4TCyuRcT5H0WzaHIiMQU5QYdrMEPZfrnKk5Tkr8juZBruWR\nqHQt/9FMAfbr2spzs6kuBX6jqqfiZE4Zh0ldS1ckM8spRnS+lCJCSW56xPx9ZE0d6SkBPjR/6B/o\n8ZOzSQoI2/Z38OkzK+NmzPjBc1kM55tNCggXLSzlhS37eHbTXp5aV8+3n9jAmScUc90ZlTHHVrjx\nmBNLcznBVTBe23Ov4tfjYlcRPLF2T6RP1AfnlcT4+ofDU1LOtf39CGeV5CACKUn9s9GLF5URDGuM\nK/HiqrJIptpFCwentYoISyoLWVnrJCvsONBJbnoy+ZlDK+4lrutmxYCU45r6VmaX9CuKktx0Tp9Z\nxGPVdZHGiHN9pPC+U0rddNwXt+znv55/myfX1vOFs2excLp/ZRQPr3HjwBjJaCAikYW1Brp9i7PS\nSE0ORNZZKXGtD08ZFGSmxE0oGI4peU7s6fVtTTyypo7mzr7Ib82b1Nz7+k7HtXwIiqS8KCsmBbix\nvWdQsk6i8XsnkkVkKk5LkidGOtgYnmAozPk/e4nvPOmkrDa2xRY3ebUku5o6eWR1HWfPmxKTAz+Q\ntOQk5k3NZd7UXP71Q7MPW64zji92/p5QPOxxlyyaRl9Iue6eVfzzvavJTE3mh5cvHDQbnunm6v9j\nlEvtoqpSUpMCkffymFWSw9ypuTxavYfnNzt9opb5cGtFc/ni6aQmBXwHajNSk5gzJZez55VEZqNO\nR91s3j1rUsSVOHdqLvPLcjl9ZhHThsh6WlJRwN7WHnY3d1F7oIPK4qxhXTxF2WnMnJQV8ZOD06hw\ne2PHIN/4sqoyag908uBKp1HErKOgSGZOyqY3FOZjd6/gB09v5tTKQm5498yRTxwBr5V8a1dfXIvk\naLNoRgH5mSkxLllwrOTSvHSqdzqB9ZIBFsmhZmyBE3sqL8zkodW7+Zc/vImIY+EDVBRlkpoc4On1\nDY5reYg4UzwqI5lbjiX86d+sYtkvXklYsW08/P4nv42Txvuyqq4UkeOAcdGaZDTYsKeVls4+Hn9z\nD9+4YB4HOnqYX9Y/C52cm87a3S3c/GA1ARG+5EM5/N+1S0hNDgzpdvDDgmn5vPSl90YyV4ZiXmku\nf/nCWZGUy8ri7LhmflF2Gs/963sozet3B0zNy+D5W97D5Dj+5YurSvnen2v45XNbKcpK5czjh1do\nA7n29ArOnT/lkHzX933q1EhvJXBmqb//zLtITopVAr+77tRhM568GeSK7U3UHugYMsYUzdLKQp5w\nO/QmBYSt+9oJK4Oqz8+ZP4WvP7qeZzbupSw/Y1BsIRFcdvI0ZpfkEAw7haInleWT5CPjayTyMlLY\nsKeP9t4guWNAkdz8gRP45BkVcV25ZQUZkbRaT5F4FsmhZmx5PPiZd0XWds/PTIm43JKTAswqyWZ9\nXSsnz8iPycocif46ok6m5LazxlV+X3tkHb/88MnDTmiOFL7+k6r6B+APUa+3AZclSqjxjlcd3NYd\n5LmafRxo7421SHLS2dXUxa6mLm6/qmrEgR0Ob4YUDz/vBURcVSMRbwY/VH3ARVWlfH95DWt3H+QT\np1f4DjZ6BAIyYu3BQAriKMB42/Izhx84ZpfkkJuezCtvN1LX3MUli4ZObPBYUlHI/St2sbmhjXml\nuf2dAQZYHLnpKXxg7mSeWtcwqBAwUSQF5B27seKRl5HC3tZuVPsr3UeT9JSkIQdtbwIk0q84vCD8\noWZseZTkpkeU0kDmTMllfV1r3Ky44SjNy4ikADe0diMCnzi9gv99pZY/vLGbKxYPbFN45PEbbE8X\nkRtF5L9E5G7vkWjhxisrtjcxvTCD4uw07nm1lmBYYxSBV0tyyaKyYVN2xxtT8zI4zV3q1WuJcqwQ\nCAiLKwpZvr7BSf2NE9wdiGfFrHJXMVy3u4W05EDcc73vgZ9q57FMbkYyXkx4LLi2hqM0v19peJOa\n6G1HGm+S4Kd+JJpAQJhRmMn2xo5IM9dvnD+Pdx1XxH88viGS/JFI/E75fgtMAT6E09J9GjB2FsE4\nhlBVVu1o5tTKIi5cOJXXtjmDSLSp/N45k7l0URnfXnbiaIk5anz+/SfwsXeVD+oHdiywpKKQTjeB\nomKIws9ophVkMDXPyfbaVN/K/St28YG5JXEtsffOnswVi2Pbsh+LRGcOjnawfSQ868Ob2AFkpyVz\n/VnHxS2SfaecM38Kl55cxukzD82lC06G5MtvNUaauQYCwo+uWEh5UVZkhcxE4ndKcLyqXi4iy1T1\nHhG5D3gpkYKNV97e305TR2+kCOl/X6kFYFLUDGdWSQ4/vrJqlCQcXd41s4h3uV17jzWWRhVxVhSN\nrEhEHCvmtW0H2LJ3DXmZKUNOHlKTA9z2jwuPmKyjRXR8Z6xbJF48pGRA/dbX3BUfjzTTCjL58RWH\n97uvKMrkr5tCkWau4FhPT33+jKMSI/HdIsX92+KuXpgHTB7meGMIvAWWllQUsiCqZUnxIRY3GWOP\n+WV5pCYHyE1PpmCY1N9ollYUsL+thy172/nR5QuPWKxrrHJMWSSuIpk8RExjLOFZwB+YOzlGWR8N\nJQL+FcmdbkX7N3A6+G4EbkuYVOOYldv7V5cTES47uYzkgAya9RjHHmnJSZxaWRi3hclQeNbXdWdU\nctassb3cwZEgL0rBjoWsreGYmp9ORkqSr3jXaOMlaFx28shJHonAb9bWr92nLwCDezAYvllRG7u6\n3GfePZMPnjgl5gdmHLv89MoqQoewPPHxk3P4801nxl1HfTwS69oa29/5tOQknrrJWRt9rLO4vIDl\nN5/pa+GyRDCsIhGRLw63X1WtTcohUH+wi93NXXzyH/orwFOSAhNmEJkIHI5ram6clifjlVjX1ti2\nSIAhu2WPNbxu1qPFSP9JG+GOIH4X7DGM8UpuhjPkpCYFDqnozhjbDKtIVPVbR0uQicD6uoOkJQd8\nt/M2jPFGRkoSKUlyTFgjhn/8FiTeE71Gu4gUWEHioVPT0MaskpxDrtg2jPGCiJCXkTImqtqNI4ff\nEW1B9FK37kqGixIj0vhlU31b3LUkDGMikZueYhbJOMOvIgm46b8AuOuS2DfhEGhs76Gxveeo9Uoy\njLHK5Ny0mAJc49jHrzL4EfCqiHiNGy8HvpsYkcYnm6OWUDWMicyPrqgi6SgVyhlHB791JL8RkVXA\n+9xNl6rqxsSJNf6IrMVtgXZjglN2iB2ajbGPb/eUqzhMeRwmNfWtFGenJaRrqGEYxmhi6UNHic17\n2yw+YhjGuMQUyVEgFFY2N5giMQxjfOK3juRWP9uM+NQe6KAnGLbUX8MwxiV+LZKz42w790gKMp7x\nMrYmUk8lwzAmDsMqEhH5JxFZB8wRkbVRj+3AupEuLiLniMhmEdkqIl+Js79ARB5xr7nCXevE23eT\niKwXkQ0icvOA8z4nIjXuvjHfzr6mvpWAwPGTs0dbFMMwjCPOSFlb9wF/Br4HRCuCNlVtGu5EEUkC\nfoljzewGVorI4wPShr8GVKvqJSIyxz3+/a5C+TSwFOgFlovIE6q6VUTeCywDFqpqj4iM+QW2ahra\nqCzOsiZ1hmGMS4a1SFT1oKrWArcDTaq6Q1V3AEEROXWEay8FtqrqNlXtBR7AUQDRzAP+5r5XDVAh\nIiXAXOB1Ve1U1SDOOiiXuuf8E/B9Ve1xz9vn87OOGjUNbcwxt5ZhGOMUvzGS/wbao163u9uGowzY\nFfV6t7stmjdxFYSILAXKgWnAeuBMESkSkUzgPGC6e84sd9/rIvKCiCzx+RlGheaOXnY2dTLPFIlh\nGOMUvwWJotq/7JuqhkXkSPTa+j5wu4hU48Rc1gAhVd3kZoU9A3QA1UAoSuZC4DRgCfB7ETkuWj4A\nEbkeuB5gxowZR0DUw2PVjv412g3DMMYjfi2SbSLyeRFJcR83AdtGOKeOfisCHEujLvoAVW1V1WtV\ntQr4GDDJu66q3qWqp6jqWUAzsMU9bTfwsDqsAMJA8cA3V9U7VXWxqi6eNGn01sJeWdtEalKABdPy\nRk0GwzCMROJXkdwAnI6jCHYDp+LO9odhJXCCiFSKSCpwFfB49AEiku/uA/gU8KKqtrr7Jrt/Z+C4\nv+5zj3sUeK+7bxaQCjT6/BxHnRXbm1g4Pc8C7YZhjFv8Nm3ch6MIfKOqQRH5LPA0kATcraobROQG\nd/8dOEH1e0REgQ3AdVGXeEhEioA+4Mao9VDuBu4WkfU4GV0fH+jWGit09gZZX3eQT5913GiLYhiG\nkTB8KRJ35v/fQImqzheRBcBFqvqd4c5T1aeApwZsuyPq+as4wfN45545xPZe4KN+5B5tqne1EAwr\nSy0+YhjGOMava+tXwFdxrANUdS2HaKFMRFZub0YETi4vGPlgwzCMYxS/iiTTDWxHEzzSwow3VtY2\nMbskhzxbn9owjHGMX0XSKCIzAQUQkX8E6hMm1TggGAqzemczSyvNrWUYxvjGby3IjcCdOD236oDt\nwEcSJtU4YMOeVjp7Q1Y/YhjGuGdERSIiAWCxqn5ARLKAgKq2JV60Y5uVtU4rMlMkhmGMd0Z0balq\nGPiS+7zDlIg/VtY2Mb0wgyl56aMtimEYRkLxGyP5q4j8q4hMF5FC75FQyY5hVJVVtc1mjRiGMSHw\nGyO50v17Y9Q2BazSDmhs7+Htfe2celwRAG/v7+BAR6/VjxiGMSEY0SJxYyQfVdXKAQ9TIi53v7yd\nq3/1GntbuwFY5cVHLGPLMIwJgN8YyS+OgizHLPvaeggr/OnNPQCsqG2iKCuV44qzRlkywzCMxOM3\nRvKsiFwmIpJQaY5Rmjt6AXi02mluvLK2icUVBdjtMgxjIuBXkXwG+APQKyKtItImIq0JlOuYoqnT\nUSTr61p5ZWsju5q6LNBuGMaEwZciUdUcVQ2oaoqq5rqvbck/l+aOXk47rpCAwDcf3wBgFe2GYUwY\nfK9yKCIXAWe5L59X1ScSI9Kxx4GOXt4zezLJgQAvb20kMzXJltY1DGPC4MsiEZHvAzcBG93HTSLy\nvUQKNtZ4c1cL3X2hQdv7QmHauoMUZKayrKoUgJNnFJCc5NdraBiGcWzjd7Q7DzhbVe9W1buBc4Dz\nEyfW2KK9J8il//13/vDG7kH7mt34SGF2KufMn0J+ZgrvmT16S/sahmEcbXy7toB8oMl9PqEWIO/s\nDRIKKwfaewbta+7oA6AwM5Wc9BRe+fL7yLBldQ3DmED4VSTfA9aIyHOA4MRKvpIwqcYYwZCzkm97\n9+AlWA50OMqlIMtZcyQr7VB0s2EYxrGP3zXb7xeR54El7qYvq2pDwqQaY0QUSc9gRRKxSLJSj6pM\nhmEYYwW/wfZLgE5VfVxVHwe6ReTixIo2dugLhwFoi6NIvBoSUySGYUxU/Abbv6mqB70XqtoCfDMx\nIo09+kKOIonn2vKq2gsyTZEYhjEx8atI4h03YYIBw7m2mjp6yUlPJsXSfQ3DmKD4Hf1WiciPRWSm\n+/gx8EYiBRtLDGeRNHX0UmRuLcMwJjB+FcnngF7gQeABoJvYtUnGNcHwMMH2zl4KTJEYhjGB8Zu1\n1cEESvcdiGeRtHX3Ddp3oL2XqbacrmEYE5iEOvZF5BwR2SwiW0VkkCISkQIReURE1orIChGZH7Xv\nJhFZLyIbROTmqO3/ISJ1IlLtPs5L5GcA6IuKkahqzD6zSAzDmOgkTJGISBLwS+BcYB5wtYjMG3DY\n14BqVV0AfAy43T13PvBpYCmwELhARI6POu8nqlrlPp5K1GfwCLoWSVihK6rflqpajMQwjAnPsIpE\nRG51/15+GNdeCmxV1W2q2osTW1k24Jh5wN8AVLUGqBCREmAu8LqqdqpqEHgBuPQwZDgieBYJxAbc\nu/pC9ATDZpEYhjGhGckiOc9dFfGrh3HtMmBX1Ovd7rZo3sRVECKyFCgHpgHrgTNFpEhEMnGaRk6P\nOu9zrjvsbhEpOAzZDomgW5AIsUWJB9rdYkSrITEMYwIzkiJZDjQDC6JXRjyCKyR+H8gXkWqczLA1\nQEhVNwG3As+4MlQDnk/pv4HjgCqgHvhRvAuLyPUiskpEVu3fv/8dCekF2yHWIvE6/5pFYhjGRGZY\nRaKqt6hqPvBk9MqIPldIrCPWipjmbou+fquqXquqVTgxkknANnffXap6iqqehaPMtrjb96pqSFXD\nwK9wXGjxZL9TVRer6uJJk95ZW/cY11aURdLUYe1RDMMw/C61u0xESkTkAvfhZ2ReCZwgIpUikgpc\nBTwefYCI5Lv7AD4FvKiqre6+ye7fGTjur/vc11OjLnEJjhssoQSjFElbHIvEFIlhGBMZX3UkbrD9\nh8DzOG3kfy4it6jqH4c6R1WDIvJZ4GkgCbhbVTeIyA3u/jtwgur3iIgCG4Droi7xkIgUAX3AjW5/\nL4DbRKQKUKAW+IzfD3u4RMdI2i1GYhiGEYPffllfB5ao6j4A1yL5KzCkIgFwU3OfGrDtjqjnrwKz\nhjj3zCG2X+NT5iNGbNZWf1Fic2cvSQEhN2PCtB0zDMMYhO+mjZ4ScTlwCOce88QE22NiJH0UZKbi\nJLYZhmFMTPxOpZeLyNPA/e7rKxlgaYxnvILEpIDEpP82d/RS6K6MaBiGMVHx22vrFhG5FDjD3XSn\nqj6SOLHGFp5rKz8jJSb9t6mj19YhMQxjwuPbua+qDwMPJ1CWMUswHCYpIOSkJ8e6tjp7mVWSPYqS\nGYZhjD4TJs7xTgiGlJQkITs9ObYg0SwSwzAMUyR+6A2FSQkEyE5LjsRIwmGlubPXakgMw5jw+FYk\nIpIqIgtE5KSoIsIJQTCkJCcJ2Wn9MZLGjh7CCsXZaaMsnWEYxujityDxfOAO4G2cgsRKEfmMqv45\nkcKNFYLhMMlJgZgYyZ6WbgDK8jNGUzTDMIxRx2+w/UfAe1V1K4CIzASeBCaEIukLKSkBITutX5HU\nNXcBUFZgisQwjImNX9dWm6dEXLYBbQmQZ0zSFwqTkhyICbbvaXEUSalZJIZhTHCGtUjc2hGAVSLy\nFPB7nB5Xl+M0ZZwQBENKsmuR9IbC9ARD1LV0kZOWTF6GFSQahjGxGcm1dWHU873Au93n+4EJMxXv\nC4VJcWMk4KxJUtfSZdaIYRgGIygSVb32aAkylgmGvawtV5H0BKlr7qI0P32UJTMMwxh9/GZtTQI+\nDVREn6Oqn0yMWGOLvlCYZLeOBJw1SfYc7OLk8vxRlswwDGP08Zu19RjwEk7r+NAIx447+kJhUpOc\nYDvAvrZuWjr7zLVlGIaBf0WSqapfTqgkY5hgSElNDpCT5gTWNze0A1ZDYhiGAf7Tf58QkfMSKskY\npi+sJEdZJFv2OpnPpkgMwzD8K5KbcJRJl4i0ikibiLQmUrCxRDAUjhQkAtQ0OIrEXFuGYRj+1yPJ\nSbQgYxmn+29/+u/b+9pJCggluZa1ZRiGMaxFIiIVI+wXEZl2JAUai/SFwiQnCWnJAZIDQm8ozJTc\ndJICtsSuYRjGSBbJD0QkgJO19QZOIWI6cDzwXuD9wDeB3YkUcrTpCzsFiSLOmiQtnX0WHzEMw3AZ\nqSDxchGZB3wE+CQwFegENuGs2f5dVe1OuJSjjNciBSA7zVUk1qzRMAwD8BEjUdWNwL8dBVnGLH0h\nJ2sLiATcrardMAzDwVZI9IFTkOhYJF7AvSw/czRFMgzDGDOYIvFBMBQ2i8QwDGMITJH4oM9t2giQ\nne5Ut1uw3TAMw8GXInHTfD8qIv/uvp4hIkt9nHeOiGwWka0i8pU4+wtE5BERWSsiK0RkftS+m0Rk\nvYhsEJGb45z7LyKiIlLs5zO8E5yCxIEWiSkSwzAM8G+R/BfwLuBq93Ub8MvhThCRJPeYc4F5wNVu\nBlg0XwOqVXUB8DHgdvfc+TjdhpcCC4ELROT4qGtPBz4I7PQp/2ETCithJWKRLJyWx5KKArLS/LYp\nMwzDGN/4VSSnquqNQDeAqjYDqSOcsxTYqqrbVLUXeABYNuCYecDf3GvWABUiUgLMBV5X1U5VDQIv\nAJdGnfcT4Es4qzUmlL5QGIAUN0Zy1dIZ/OGG0xP9toZhGMcMfhVJn2thKETWJwmPcE4ZsCvq9W53\nWzRv4ioI11VWDkwD1gNnikiRiGQC5wHT3eOWAXWq+qZP2d8RwbCjq1KSrIrdMAwjHn79Mz8DHgEm\ni8h3gX8Evn4E3v/7wO0iUg2sA9YAIVXdJCK3As8AHUA1EHKVytdw3FrDIiLXA9cDzJgx47AFDLoW\nSXLA8hIMwzDi4bdp470i8gZOSxQBLlbVTSOcVodrRbhMc7dFX7cVuBacgD6wHdjm7rsLuMvd9/9w\nLJqZQCXwpnM404DVIrJUVRsGXPtO4E6AxYsXH7YLrC9kFolhGMZwjKhIXJfWBlWdA9QcwrVXAieI\nSCWOArkK+PCAa+cDnW4M5VPAi65yQUQmq+o+EZmB4/46TVVbgMlR59cCi1W18RDkOiSC4dgYiWEY\nhhGLnxYpITeFd4aq+s6SUtWgiHwWeBpIAu5W1Q0icoO7/w6coPo9IqLABuC6qEs8JCJFQB9wo6tE\njjp9QcciSTZFYhiGERe/MZICYIOIrMCJWQCgqhcNd5KqPoXT3DF62x1Rz18FZg1x7pkjCaWqFSMd\n807pi1gk5toyDMOIh19F8o2ESjGGCboxEgu2G4ZhxMdvsP0Ft75jibtpharuS5xYYwevjiTZLBLD\nMIy4+G2RcgWwArgcuAJ4XUT+MZGCjRU8RZJqMRLDMIy4+HVt/RuwxLNC3ILEvwJ/TJRgYwWvINEs\nEsMwjPj4nWYHBriyDhzCucc0fVaQaBiGMSx+LZLlIvI0cL/7+krgz4kRaWwRtIJEwzCMYfEbbL9F\nRC4FznA33amqjyROrLGDV5BodSSGYRjx8aVI3Or0p1T1Yfd1hohUqGptIoUbC/QGzSIxDMMYDr/T\n7D8Q2+035G4b91iLFMMwjOHxOzomu/2wAHCfj7QeybigvyDRLBLDMIx4+FUk+0Uk0g7FXRMkYY0S\nxxIDF7YyDMMwYvGbtXUDcK+I/AKnjfwunKVxxz39C1uZIjEMw4iH36ytt4HTRCTbfd2eUKnGENYi\nxTAMY3j8tki5SURycTr//lREVovIiKsUjgciC1tZQaJhGEZc/I6On3QXnPogUARcg7NM7rgnaBaJ\nYRjGsPhVJN4oeh7wG1XdELVtXGO9tgzDMIbHryJ5Q0SewVEkT4tIDrF1JeOW3qCbtWWuLcMwjLj4\nzdq6DqgCtqlqp7sE7rWJE2vsEAyHSQoIAasjMQzDiIvfrK0wsDrq9QGcDsDjnmBIrRjRMAxjGMxf\nMwJ9IbUaEsMwjGGwEXIEguGwBdoNwzCGwW+MBBFJAkqiz1HVnYkQaizRFwqbRWIYhjEMftvIfw74\nJoqQ+iUAABOXSURBVLCX/mwtBRYkSK4xQ19ISbEYiWEYxpD4tUhuAma7QfYJRTAUtkWtDMMwhsHv\nCLkLOJhIQcYqfWG1GIlhGMYw+FUk24DnReSrIvJF7zHSSSJyjohsFpGtIvKVOPsLROQREVkrIitE\nZH7UvptEZL2IbBCRm6O2/6d7fLWIPCMipT4/w2ERDIVJNYvEMAxjSPyOkDuBv+AsZpUT9RgSNzj/\nS+BcYB5wtYjMG3DY14BqVV2A05b+dvfc+cCngaXAQuACETnePecHqrpAVauAJ4B/9/kZDou+kFkk\nhmEYw+G3IPFbAIfYRn4psFVVt7nnPgAsAzZGHTMPt/mjqtaISIWIlABzgddVtdM99wXgUuA2t3mk\nRxZO0D9h9IXCJFt7FMMwjCHx20Z+voisATYAG0TkDRE5cYTTynBiKx673W3RvImjIBCRpUA5MA1Y\nD5wpIkUikonT42t6lDzfFZFdwEdIsEUSDCkpZpEYhmEMid+p9p3AF1W1XFXLgX8BfnUE3v/7QL6I\nVAOfA9YAIVXdBNwKPAMsB6qBkHeSqv6bqk4H7gU+G+/CInK9iKwSkVX79+8/bAGDYbNIDMMwhsPv\nCJmlqs95L1T1eRy30nDUEWVF4FgaddEHqGqrql7rxjs+BkzCCeyjqnep6imqehbQDGyJ8x73ApfF\ne3NVvVNVF6vq4kmTJo0g6tD0hpSUZFMkhmEYQ+E7a0tEvuHGMCpE5Ou4A/4wrAROEJFKEUkFrgIe\njz5ARPLdfQCfAl70YiAiMtn9OwPH/XWf+/qEqEssA2p8fobDIhgKW0GiYRjGMPgtSPwk8C3gYff1\nS+62IVHVoIh8FngaSALuVtUNInKDu/8OnKD6PSKiOPGX66Iu8ZDbrr4PuFFVW9zt3xeR2TgV9juA\nG3x+hsMiaFlbhmEYw+I3a6sZ+PyhXlxVnwKeGrDtjqjnrwKzhjj3zCG2x3VlJYq+sFW2G4ZhDMew\nikREfqqqN4vIn4iTZquqFyVMsjFC0HptGYZhDMtIFslv3b8/TLQgYxXr/msYhjE8wyoSVX3DfVql\nqrdH7xORm4AXEiXYWMGpbDdFYhiGMRR+R8iPx9n2iSMox5glGA5bQaJhGMYwjBQjuRr4MFApItGp\nuzlAUyIFGys4a7abRWIYhjEUI8VI/g7UA8XAj6K2twFrEyXUWKIvFCYl2SwSwzCMoRgpRrIDp1bj\nXUdHnLFHXyhMilkkhmEYQ+K3aeNpIrJSRNpFpFdEQiLSOvKZxzbhsBJWrCDRMAxjGPxOtX8BXA28\nBWTgtDP5ZaKEGiv0hZ3l6S391zAMY2h8j5CquhVIUtWQqv4vcE7ixBobBENODWayFSQahmEMid9e\nW51uc8VqEbkNJwA/7qfpfSGzSAzDMEbC7wh5DU7jxc8CHTjt4Y9qz6vRoM+1SKyOxDAMY2j8Nm3c\n4T7t4v+3d/dBUlVnHse/P1QgooAIUgoSCGIIYQETlqgYo9FVfFnUrK5QWkYluqYwogYjrK6Fltlo\niJZu3F11NWrUaDSKIcSXYQUxcREQ5GV4FdEoLFlJZdUkrsIwz/5xTsOlnZ5756Wn+848n6quvq/d\nz9M9c0/fe+45J/QC3CHUxToSb9nunHOlpTVIXEUjY6Kb2YhWj6iKeB2Jc86lSzsjOT0+T47PhU4c\nz6eRAqa98DoS55xLl6VBIpL+xsyOSKy6VtIyYFo5g6u03XUkXpA451wpWY+QkjQ2MXN0E/bNrcIZ\niTdIdM650rLe/jsJ+ImkHoCA/yVlqN32oK7e79pyzrk0We/aWgqMjAUJZvZBWaOqEnWFMxLva8s5\n50pKu2vrfDN7RNLVRcsBMLPbyxhbxXkdiXPOpUs7I+kWn/cvdyDVaPddW35pyznnSkm7a+ue+Nxh\nGiEmeYNE55xLl3Zp618aW29mV7RuONVlhzdIdM65VGmXtpa2SRRVqs7rSJxzLlXapa2H2iqQauR1\nJM45ly7T7b+S+gDXAsOAroXlZvb1MsVVFbyLFOecS5f1CPkosBYYROj9921gSdpOksZJWi9po6RP\ndaci6QBJsyStlLRY0vDEuimSaiWtlnRlYvlMSeviPrMk9cyYQ5MVGiR6y3bnnCsta0FyoJndD+ww\nswVmdjHQ6NmIpL0Iw/GeQjiTmShpWNFm/wgsj70IXwDcGfcdDlwCjAFGAqdLOizuMxcYHvfZAEzP\nmEOTeYNE55xLl/UIuSM+b5V0mqQjgF4p+4wBNprZJjPbDjwOnFG0zTBgHoCZrQMGSuoLfAFYZGYf\nmVkdsAD4RtyuJi4DeBXonzGHJvOBrZxzLl3WguTm2D3Kd4GpwH3AVSn79APeTcxvjsuSVhALCElj\ngM8SCoZa4KuSDpS0L3AqYVTGYhcDz2XMocm8jsQ559Jl7bRxUexf6wPg+FZ8/1uAOyUtB1YBrwM7\nzWytpFuBGsLQvsuBnckdJV0H1BHqbz5F0qXApQADBgxoVnBeR+Kcc+my/tR+RVKNpEmSDsi4zxb2\nPIvoH5ftYmYfmtlFZjaKUEfSB9gU191vZl82s2MJvQ1vKOwn6ULCoFvnmVmDA2yZ2b1mNtrMRvfp\n0ydjyHvadUbidSTOOVdSpiOkmR0OXA98EVgqaY6k81N2WwIMkTRIUmdgAjA7uYGknnEdwLeAl83s\nw7juoPg8gHD562dxfhzwPWC8mX2UJf7mqttpdBJ08pbtzjlXUuaf2ma22MyuJlSi/xFotLFirBC/\nHHiBcOvwE2a2WtJlki6Lm30BqJW0nnB315TESzwlaQ3wK2Cymb0fl99F6ERyrqTlku7OmkNT7aiv\n9/oR55xLkbVBYnfgLMJZxWBgFqFAaZSZPQs8W7Ts7sT0QuDwEvt+tcTywxpaXg476swLEuecS5G1\nsn0F8AxwUzz4dwh19fVe0e6ccymyFiSfK1Wp3Z4NO7g7n+yor3QYzjlX1bIOtdvhChGACWMGMGFM\n824dds65jsIrAJxzzrWIFyTOOedaJFNBIumHkrpL2kfSi5K2ZWhH4pxzrgPIekZyUmwoeDqhC/nD\ngGvKFZRzzrn8yFqQFCrlTwOejP1uOeecc5lv/50jaR3wf8C344iJH5cvLOecc3mRta+tacDRwGgz\n20Hokbd4bBHnnHMdUNbK9nMIoyPulHQ98AhwSFkjc845lwvK0tZQ0kozGyHpGOBmYCZwg5l9pdwB\ntgZJ24DfNXG33sAfyhBOW/M8qk97ycXzqD6tnctnzSx1HI6sdSSFQaVOA+41s19LurnZobWxLB9E\nMUmvmdnocsTTljyP6tNecvE8qk+lcsl619YWSfcA5wLPSurShH2dc861Y1kLg78njCtychwXpBfe\njsQ55xzZ79r6CHgTOFnS5cBBZlZT1sgq795KB9BKPI/q015y8TyqT0VyyVrZPgW4BHg6LjqLUFfy\n4zLG5pxzLgcy37UFHGVmf4nz3YCFZjaizPE555yrclnrSMTuO7eI0+1y6EBJ4yStl7RR0rRKx5OV\npEMlzZe0RtLqeBaJpF6S5kp6Iz4fUOlYs5C0l6TXJc2J83nNo6ekX0haJ2mtpKPymIukq+LfVa2k\nxyR1zUsekn4i6T1JtYllJWOXND3+/6+XdHJlov60EnnMjH9bKyXNktQzsa7N8shakDwALJI0Q9IM\n4FXg/rJFVSGS9gL+FTgFGAZMlDSsslFlVgd818yGAUcCk2Ps04AXzWwI8GKcz4MpwNrEfF7zuBN4\n3syGAiMJOeUqF0n9gCsIPVsMB/YCJpCfPB4ExhUtazD2+D8zAfhi3Off4nGhGjzIp/OYCwyPV4c2\nANOh7fPIWtl+O3AR8Mf4uMjM7ihXUBU0BthoZpvMbDvwODnpCsbMtprZsjj9J8IBqx8h/ofiZg8B\nZ1Ymwuwk9Se0WbovsTiPefQAjiX+6DKz7fGux9zlQmhz9hlJewP7Av9NTvIws5cJx62kUrGfATxu\nZp+Y2VvARsJxoeIaysPMasysLs6+CvSP022aR2qDxFiKrY6/qJaVK5Aq0Q94NzG/GchF6/0kSQOB\nI4BFQF8z2xpX/R7oW6GwmuIO4HvA/ollecxjELANeEDSSGAp4UwrV7mY2RZJPwLeIXTcWmNmNZJy\nlUeRUrH3IxyQCzbHZXlwMfDzON2meaSekZjZTmC9JB+8PAck7Qc8BVwZx5DZxcKdFel3V1SQpNOB\n98xsaalt8pBHtDfwJeDfzewIQmene1z+yUMusf7gDELBeAjQrXhguzzkUUqeYy+QdB3h8vajlXj/\nrF2kHACslrSY8M8AgJmNL0tUlbMFODQx3z8uywVJ+xAKkUfNrHCr9v9IOtjMtko6GHivchFmMhYY\nL+lUoCvQXdIj5C8PCL8CN5vZojj/C0JBkrdcTgTeMrNtAJKeJvQGnrc8kkrFnrtjgKQLCYMOnmC7\nb8Nt0zyyVrb/EyHQm4DbEo/2ZgkwRNIgSZ0JlVWzKxxTJpJEuBa/NtZpFcwGvhmnvwn8sq1jawoz\nm25m/c1sIOHzn2dm55OzPADM7PfAu5I+HxedAKwhf7m8Axwpad/4d3YCoQ4ub3kklYp9NjBBUhdJ\ng4AhwOIKxJeJpHGEy8DjY8PxgrbNw8xKPghD6o5tYPkxwODG9s3rAziVcPfDm8B1lY6nCXEfQzg9\nXwksj49TgQMJd6W8Afwn0KvSsTYhp+OAOXE6l3kAo4DX4vfyDOHsPne5ADcC64Ba4GGgS17yAB4D\ntgI7CGeJkxqLHbgu/v+vB06pdPwpeWwk1OsW/ufvrkQejTZIjPfwTzezVUXL/wr4ZzP725I7O+ec\n6xDSLm31LS5EAOKygWWJyDnnXK6kFSQ9G1n3mdYMxDnnXD6lFSSvSbqkeKGkbxHuiXfOOdfBpdWR\n9AVmAdvZXXCMBjoDZ1m4K8U551wHlrX33+OB4XF2tZnNK2tUzjnnciNrX1vzzezH8eGFiGsySSbp\ntsT81NgBaGu89oOSzm6N10p5n3NiD77zG1g3M/aOO7MZrzsqNsCsWpL+3Mz9zmxOx6fNfT9XGT7u\numsrnwDfkNS70oEkxU4Is5oEXGJmxzew7lJghJk1ZwjqUYQ2P5kpyMP/75mEnrRdO5aHP0TXPtQR\nhgG9qnhF8RlF4deopOMkLZD0S0mbJN0i6TxJiyWtkjQ48TInSnpN0obYX1dhTJOZkpbE8Rr+IfG6\nv5E0m9DSvDieifH1ayXdGpfdQGj0eX/xWUd8nf2ApZLOldRH0lPxfZdIGhu3GyNpocI4K/8l6fOx\nB4WbgHMlLY/7z5A0NfH6tZIGxsd6ST8lNAw8VNJJ8TWXSXoy9rVG/KzWxLx/1ECOX4vvtzzGs39c\nfk3i87qxoS+y1DaSLojLVkh6WNLRwHhgZnyfwfHxvKSl8TsYGvcdFPNYJenmht7XVbFKt9b0R8d4\nAH8GugNvAz2AqcCMuO5B4OzktvH5OOB94GBCS+otwI1x3RTgjsT+zxN+GA0htPrtSjhLuD5u04XQ\nwnxQfN2/AIMaiPMQQpcgfQh90c0DzozrXiKMydFgfonpnwHHxOkBhG5riPnvHadPBJ6K0xcCdyX2\nnwFMTczXEtptDQTqgSPj8t7Ay0C3OH8tcAOh1fZ6dteB9mwg3l8Re60gFIJ7AycRCnvFz3IOcGzR\nd9LgNoRxLzYAveN2vUp8ty8CQ+L0Vwhd4EDo0uOCOD05+Xn6o/ofTTmtd65FzOzD+Gv6CkJ35Fks\nsdjdt6Q3gZq4fBWQvMT0hJnVA29I2gQMJRz0RiTOdnoQCprtwGIL4zQU+2vgJdvdQeGjhAPlMxnj\nhVBIDJN2DSLaPZ4p9AAekjSE0J3NPk14zYLfmVmhe/AjCZeNXonv1RlYCHwAfEw4e5pDONgXewW4\nPeb3tJltlnQS4TN7PW6zH+HzejmxX6ltRgJPmtkfAMysePyPQs/URwNPJj6bLvF5LPB3cfph4NbU\nT8JVDS9IXFu7gzCuzQOJZXXEy6zxun/nxLpPEtP1ifl69vz7Lb790Ai/mr9jZi8kV0g6jkQv1mXQ\niXDW8HHR+94FzDezsxTGjHmpxP67Po+oa2I6GbeAuWY2sfgFJI0hdK54NnA58PXkejO7RdKvCXUz\nrygMxSrgB2Z2TyO5NbiNpO80sk9BJ+B9MxtVYn2uu3LvyLyOxLWp+Ev1CULFdcHbwJfj9Hia90v9\nHEmdYr3J5wiXdl4Avq3QvT6SDpfULeV1FgNfk9RbYVC3icCCJsZSA+w6sEoqHDh7sLsr7wsT2/+J\nPQfxepswjgmSvkS4HNeQV4Gxkg6L23aLOe4H9DCzZwl1UiOLd5Q02MxWmdmthF6vhxI+r4sT9Sz9\nJB1UtGupbeYRvoMD4/JexblZGB/nLUnnxG2kMOAXhDOkCXH6vBL5uirlBYmrhNsI1/cL/oNw8F4B\nHEXzzhbeIRQCzwGXxbOB+wiV6csk1QL3kHIWHi+jTQPmAyuApWbW1O7RrwBGx4rnNcBlcfkPgR9I\ner0ojvmES2HLJZ1LGFOml6TVhLOJDSVi3UYokB6TtJJwWWso4cA9Jy77LXB1A7tfGSvxVxJ6k33O\nzGoI9TsLJa0ijJ+SLOAotY2ZrQa+DyyI32NhKIPHgWtihf5gQiExKW6zmt1DWU8BJsfXzMuIhC7K\n1CDROeecK8XPSJxzzrWIFyTOOedaxAsS55xzLeIFiXPOuRbxgsQ551yLeEHinHOuRbwgcc451yJe\nkDjnnGuR/wfXEj4yp5rzKgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x9c8519ba90>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "rfecv_U2R = RFECV(estimator=clf_U2R, step=1, cv=10, scoring='accuracy')\n",
    "rfecv_U2R.fit(X_U2R_test, Y_U2R_test)\n",
    "# Plot number of features VS. cross-validation scores\n",
    "plt.figure()\n",
    "plt.xlabel(\"Number of features selected\")\n",
    "plt.ylabel(\"Cross validation score (nb of correct classifications)\")\n",
    "plt.title('RFECV U2R')\n",
    "plt.plot(range(1, len(rfecv_U2R.grid_scores_) + 1), rfecv_U2R.grid_scores_)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Using 13 Features for each category"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Confusion Matrices\n",
    "## DoS"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(9778, 13)"
      ]
     },
     "execution_count": 87,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# reduce test dataset to 13 features, use only features described in rfecolname_DoS etc.\n",
    "X_DoS_test2=X_DoS_test[:,rfecolindex_DoS]\n",
    "X_Probe_test2=X_Probe_test[:,rfecolindex_Probe]\n",
    "X_R2L_test2=X_R2L_test[:,rfecolindex_R2L]\n",
    "X_U2R_test2=X_U2R_test[:,rfecolindex_U2R]\n",
    "X_U2R_test2.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>Predicted attacks</th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Actual attacks</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>9602</td>\n",
       "      <td>109</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2625</td>\n",
       "      <td>4835</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Predicted attacks     0     1\n",
       "Actual attacks               \n",
       "0                  9602   109\n",
       "1                  2625  4835"
      ]
     },
     "execution_count": 93,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Y_DoS_pred2=clf_rfeDoS.predict(X_DoS_test2)\n",
    "# Create confusion matrix\n",
    "pd.crosstab(Y_DoS_test, Y_DoS_pred2, rownames=['Actual attacks'], colnames=['Predicted attacks'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Probe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>Predicted attacks</th>\n",
       "      <th>0</th>\n",
       "      <th>2</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Actual attacks</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>8709</td>\n",
       "      <td>1002</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>944</td>\n",
       "      <td>1477</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Predicted attacks     0     2\n",
       "Actual attacks               \n",
       "0                  8709  1002\n",
       "2                   944  1477"
      ]
     },
     "execution_count": 92,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Y_Probe_pred2=clf_rfeProbe.predict(X_Probe_test2)\n",
    "# Create confusion matrix\n",
    "pd.crosstab(Y_Probe_test, Y_Probe_pred2, rownames=['Actual attacks'], colnames=['Predicted attacks'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## R2L"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>Predicted attacks</th>\n",
       "      <th>0</th>\n",
       "      <th>3</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Actual attacks</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>9649</td>\n",
       "      <td>62</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2560</td>\n",
       "      <td>325</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Predicted attacks     0    3\n",
       "Actual attacks              \n",
       "0                  9649   62\n",
       "3                  2560  325"
      ]
     },
     "execution_count": 94,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Y_R2L_pred2=clf_rfeR2L.predict(X_R2L_test2)\n",
    "# Create confusion matrix\n",
    "pd.crosstab(Y_R2L_test, Y_R2L_pred2, rownames=['Actual attacks'], colnames=['Predicted attacks'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## U2R"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>Predicted attacks</th>\n",
       "      <th>0</th>\n",
       "      <th>4</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Actual attacks</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>9706</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>52</td>\n",
       "      <td>15</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Predicted attacks     0   4\n",
       "Actual attacks             \n",
       "0                  9706   5\n",
       "4                    52  15"
      ]
     },
     "execution_count": 95,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Y_U2R_pred2=clf_rfeU2R.predict(X_U2R_test2)\n",
    "# Create confusion matrix\n",
    "pd.crosstab(Y_U2R_test, Y_U2R_pred2, rownames=['Actual attacks'], colnames=['Predicted attacks'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Cross Validation: Accuracy, Precision, Recall, F-measure"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## DoS"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.99732 (+/- 0.00251)\n",
      "Precision: 0.99679 (+/- 0.00464)\n",
      "Recall: 0.99705 (+/- 0.00356)\n",
      "F-measure: 0.99692 (+/- 0.00288)\n"
     ]
    }
   ],
   "source": [
    "accuracy = cross_val_score(clf_rfeDoS, X_DoS_test2, Y_DoS_test, cv=10, scoring='accuracy')\n",
    "print(\"Accuracy: %0.5f (+/- %0.5f)\" % (accuracy.mean(), accuracy.std() * 2))\n",
    "precision = cross_val_score(clf_rfeDoS, X_DoS_test2, Y_DoS_test, cv=10, scoring='precision')\n",
    "print(\"Precision: %0.5f (+/- %0.5f)\" % (precision.mean(), precision.std() * 2))\n",
    "recall = cross_val_score(clf_rfeDoS, X_DoS_test2, Y_DoS_test, cv=10, scoring='recall')\n",
    "print(\"Recall: %0.5f (+/- %0.5f)\" % (recall.mean(), recall.std() * 2))\n",
    "f = cross_val_score(clf_rfeDoS, X_DoS_test2, Y_DoS_test, cv=10, scoring='f1')\n",
    "print(\"F-measure: %0.5f (+/- %0.5f)\" % (f.mean(), f.std() * 2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Probe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.99085 (+/- 0.00559)\n",
      "Precision: 0.98674 (+/- 0.01180)\n",
      "Recall: 0.98467 (+/- 0.01027)\n",
      "F-measure: 0.98565 (+/- 0.00872)\n"
     ]
    }
   ],
   "source": [
    "accuracy = cross_val_score(clf_rfeProbe, X_Probe_test2, Y_Probe_test, cv=10, scoring='accuracy')\n",
    "print(\"Accuracy: %0.5f (+/- %0.5f)\" % (accuracy.mean(), accuracy.std() * 2))\n",
    "precision = cross_val_score(clf_rfeProbe, X_Probe_test2, Y_Probe_test, cv=10, scoring='precision_macro')\n",
    "print(\"Precision: %0.5f (+/- %0.5f)\" % (precision.mean(), precision.std() * 2))\n",
    "recall = cross_val_score(clf_rfeProbe, X_Probe_test2, Y_Probe_test, cv=10, scoring='recall_macro')\n",
    "print(\"Recall: %0.5f (+/- %0.5f)\" % (recall.mean(), recall.std() * 2))\n",
    "f = cross_val_score(clf_rfeProbe, X_Probe_test2, Y_Probe_test, cv=10, scoring='f1_macro')\n",
    "print(\"F-measure: %0.5f (+/- %0.5f)\" % (f.mean(), f.std() * 2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## R2L"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.97451 (+/- 0.00906)\n",
      "Precision: 0.96683 (+/- 0.01316)\n",
      "Recall: 0.96069 (+/- 0.01547)\n",
      "F-measure: 0.96367 (+/- 0.01300)\n"
     ]
    }
   ],
   "source": [
    "accuracy = cross_val_score(clf_rfeR2L, X_R2L_test2, Y_R2L_test, cv=10, scoring='accuracy')\n",
    "print(\"Accuracy: %0.5f (+/- %0.5f)\" % (accuracy.mean(), accuracy.std() * 2))\n",
    "precision = cross_val_score(clf_rfeR2L, X_R2L_test2, Y_R2L_test, cv=10, scoring='precision_macro')\n",
    "print(\"Precision: %0.5f (+/- %0.5f)\" % (precision.mean(), precision.std() * 2))\n",
    "recall = cross_val_score(clf_rfeR2L, X_R2L_test2, Y_R2L_test, cv=10, scoring='recall_macro')\n",
    "print(\"Recall: %0.5f (+/- %0.5f)\" % (recall.mean(), recall.std() * 2))\n",
    "f = cross_val_score(clf_rfeR2L, X_R2L_test2, Y_R2L_test, cv=10, scoring='f1_macro')\n",
    "print(\"F-measure: %0.5f (+/- %0.5f)\" % (f.mean(), f.std() * 2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## U2R"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.99652 (+/- 0.00319)\n",
      "Precision: 0.87747 (+/- 0.15709)\n",
      "Recall: 0.89183 (+/- 0.17196)\n",
      "F-measure: 0.87497 (+/- 0.11358)\n"
     ]
    }
   ],
   "source": [
    "accuracy = cross_val_score(clf_rfeU2R, X_U2R_test2, Y_U2R_test, cv=10, scoring='accuracy')\n",
    "print(\"Accuracy: %0.5f (+/- %0.5f)\" % (accuracy.mean(), accuracy.std() * 2))\n",
    "precision = cross_val_score(clf_rfeU2R, X_U2R_test2, Y_U2R_test, cv=10, scoring='precision_macro')\n",
    "print(\"Precision: %0.5f (+/- %0.5f)\" % (precision.mean(), precision.std() * 2))\n",
    "recall = cross_val_score(clf_rfeU2R, X_U2R_test2, Y_U2R_test, cv=10, scoring='recall_macro')\n",
    "print(\"Recall: %0.5f (+/- %0.5f)\" % (recall.mean(), recall.std() * 2))\n",
    "f = cross_val_score(clf_rfeU2R, X_U2R_test2, Y_U2R_test, cv=10, scoring='f1_macro')\n",
    "print(\"F-measure: %0.5f (+/- %0.5f)\" % (f.mean(), f.std() * 2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Stratified CV => Stays the same"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.99732 (+/- 0.00251)\n"
     ]
    }
   ],
   "source": [
    "from sklearn.model_selection import StratifiedKFold\n",
    "accuracy = cross_val_score(clf_rfeDoS, X_DoS_test2, Y_DoS_test, cv=StratifiedKFold(10), scoring='accuracy')\n",
    "print(\"Accuracy: %0.5f (+/- %0.5f)\" % (accuracy.mean(), accuracy.std() * 2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.99085 (+/- 0.00559)\n"
     ]
    }
   ],
   "source": [
    "accuracy = cross_val_score(clf_rfeProbe, X_Probe_test2, Y_Probe_test, cv=StratifiedKFold(10), scoring='accuracy')\n",
    "print(\"Accuracy: %0.5f (+/- %0.5f)\" % (accuracy.mean(), accuracy.std() * 2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.97451 (+/- 0.00906)\n"
     ]
    }
   ],
   "source": [
    "accuracy = cross_val_score(clf_rfeR2L, X_R2L_test2, Y_R2L_test, cv=StratifiedKFold(10), scoring='accuracy')\n",
    "print(\"Accuracy: %0.5f (+/- %0.5f)\" % (accuracy.mean(), accuracy.std() * 2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.99652 (+/- 0.00319)\n"
     ]
    }
   ],
   "source": [
    "accuracy = cross_val_score(clf_rfeU2R, X_U2R_test2, Y_U2R_test, cv=StratifiedKFold(10), scoring='accuracy')\n",
    "print(\"Accuracy: %0.5f (+/- %0.5f)\" % (accuracy.mean(), accuracy.std() * 2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# CV 2, 5, 10, 30, 50 fold"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## DoS"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.99662 (+/- 0.00116)\n"
     ]
    }
   ],
   "source": [
    "accuracy = cross_val_score(clf_rfeDoS, X_DoS_test2, Y_DoS_test, cv=2, scoring='accuracy')\n",
    "print(\"Accuracy: %0.5f (+/- %0.5f)\" % (accuracy.mean(), accuracy.std() * 2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.99703 (+/- 0.00057)\n"
     ]
    }
   ],
   "source": [
    "accuracy = cross_val_score(clf_rfeDoS, X_DoS_test2, Y_DoS_test, cv=5, scoring='accuracy')\n",
    "print(\"Accuracy: %0.5f (+/- %0.5f)\" % (accuracy.mean(), accuracy.std() * 2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.99732 (+/- 0.00251)\n"
     ]
    }
   ],
   "source": [
    "accuracy = cross_val_score(clf_rfeDoS, X_DoS_test2, Y_DoS_test, cv=10, scoring='accuracy')\n",
    "print(\"Accuracy: %0.5f (+/- %0.5f)\" % (accuracy.mean(), accuracy.std() * 2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.99726 (+/- 0.00390)\n"
     ]
    }
   ],
   "source": [
    "accuracy = cross_val_score(clf_rfeDoS, X_DoS_test2, Y_DoS_test, cv=30, scoring='accuracy')\n",
    "print(\"Accuracy: %0.5f (+/- %0.5f)\" % (accuracy.mean(), accuracy.std() * 2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.99703 (+/- 0.00599)\n"
     ]
    }
   ],
   "source": [
    "accuracy = cross_val_score(clf_rfeDoS, X_DoS_test2, Y_DoS_test, cv=50, scoring='accuracy')\n",
    "print(\"Accuracy: %0.5f (+/- %0.5f)\" % (accuracy.mean(), accuracy.std() * 2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Probe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.99060 (+/- 0.00165)\n"
     ]
    }
   ],
   "source": [
    "accuracy = cross_val_score(clf_rfeProbe, X_Probe_test2, Y_Probe_test, cv=2, scoring='accuracy')\n",
    "print(\"Accuracy: %0.5f (+/- %0.5f)\" % (accuracy.mean(), accuracy.std() * 2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.99102 (+/- 0.00230)\n"
     ]
    }
   ],
   "source": [
    "accuracy = cross_val_score(clf_rfeProbe, X_Probe_test2, Y_Probe_test, cv=5, scoring='accuracy')\n",
    "print(\"Accuracy: %0.5f (+/- %0.5f)\" % (accuracy.mean(), accuracy.std() * 2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.99085 (+/- 0.00559)\n"
     ]
    }
   ],
   "source": [
    "accuracy = cross_val_score(clf_rfeProbe, X_Probe_test2, Y_Probe_test, cv=10, scoring='accuracy')\n",
    "print(\"Accuracy: %0.5f (+/- %0.5f)\" % (accuracy.mean(), accuracy.std() * 2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.99151 (+/- 0.00731)\n"
     ]
    }
   ],
   "source": [
    "accuracy = cross_val_score(clf_rfeProbe, X_Probe_test2, Y_Probe_test, cv=30, scoring='accuracy')\n",
    "print(\"Accuracy: %0.5f (+/- %0.5f)\" % (accuracy.mean(), accuracy.std() * 2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.99110 (+/- 0.01073)\n"
     ]
    }
   ],
   "source": [
    "accuracy = cross_val_score(clf_rfeProbe, X_Probe_test2, Y_Probe_test, cv=50, scoring='accuracy')\n",
    "print(\"Accuracy: %0.5f (+/- %0.5f)\" % (accuracy.mean(), accuracy.std() * 2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## R2L"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.97134 (+/- 0.00174)\n"
     ]
    }
   ],
   "source": [
    "accuracy = cross_val_score(clf_rfeR2L, X_R2L_test2, Y_R2L_test, cv=2, scoring='accuracy')\n",
    "print(\"Accuracy: %0.5f (+/- %0.5f)\" % (accuracy.mean(), accuracy.std() * 2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.97380 (+/- 0.00588)\n"
     ]
    }
   ],
   "source": [
    "accuracy = cross_val_score(clf_rfeR2L, X_R2L_test2, Y_R2L_test, cv=5, scoring='accuracy')\n",
    "print(\"Accuracy: %0.5f (+/- %0.5f)\" % (accuracy.mean(), accuracy.std() * 2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.97451 (+/- 0.00906)\n"
     ]
    }
   ],
   "source": [
    "accuracy = cross_val_score(clf_rfeR2L, X_R2L_test2, Y_R2L_test, cv=10, scoring='accuracy')\n",
    "print(\"Accuracy: %0.5f (+/- %0.5f)\" % (accuracy.mean(), accuracy.std() * 2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.97459 (+/- 0.01652)\n"
     ]
    }
   ],
   "source": [
    "accuracy = cross_val_score(clf_rfeR2L, X_R2L_test2, Y_R2L_test, cv=30, scoring='accuracy')\n",
    "print(\"Accuracy: %0.5f (+/- %0.5f)\" % (accuracy.mean(), accuracy.std() * 2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 122,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.97531 (+/- 0.01820)\n"
     ]
    }
   ],
   "source": [
    "accuracy = cross_val_score(clf_rfeR2L, X_R2L_test2, Y_R2L_test, cv=50, scoring='accuracy')\n",
    "print(\"Accuracy: %0.5f (+/- %0.5f)\" % (accuracy.mean(), accuracy.std() * 2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## U2R"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.99601 (+/- 0.00021)\n"
     ]
    }
   ],
   "source": [
    "accuracy = cross_val_score(clf_rfeU2R, X_U2R_test2, Y_U2R_test, cv=2, scoring='accuracy')\n",
    "print(\"Accuracy: %0.5f (+/- %0.5f)\" % (accuracy.mean(), accuracy.std() * 2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.99714 (+/- 0.00153)\n"
     ]
    }
   ],
   "source": [
    "accuracy = cross_val_score(clf_rfeU2R, X_U2R_test2, Y_U2R_test, cv=5, scoring='accuracy')\n",
    "print(\"Accuracy: %0.5f (+/- %0.5f)\" % (accuracy.mean(), accuracy.std() * 2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 125,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.99652 (+/- 0.00319)\n"
     ]
    }
   ],
   "source": [
    "accuracy = cross_val_score(clf_rfeU2R, X_U2R_test2, Y_U2R_test, cv=10, scoring='accuracy')\n",
    "print(\"Accuracy: %0.5f (+/- %0.5f)\" % (accuracy.mean(), accuracy.std() * 2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 126,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.99683 (+/- 0.00643)\n"
     ]
    }
   ],
   "source": [
    "accuracy = cross_val_score(clf_rfeU2R, X_U2R_test2, Y_U2R_test, cv=30, scoring='accuracy')\n",
    "print(\"Accuracy: %0.5f (+/- %0.5f)\" % (accuracy.mean(), accuracy.std() * 2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 127,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.99652 (+/- 0.00692)\n"
     ]
    }
   ],
   "source": [
    "accuracy = cross_val_score(clf_rfeU2R, X_U2R_test2, Y_U2R_test, cv=50, scoring='accuracy')\n",
    "print(\"Accuracy: %0.5f (+/- %0.5f)\" % (accuracy.mean(), accuracy.std() * 2))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
